Method and apparatus for annotating media streams

ABSTRACT

In one embodiment, the invention is a method and apparatus for annotating media streams. One embodiment of a method for facilitating creation of an annotated media stream by a user includes receiving the media stream from the user, mapping at least one item of supplemental content to at least a portion of the media stream to produce the annotated media stream, and storing the annotated media stream. In another embodiment, a method for distributing an annotated media stream provided by a user includes receiving the annotated media stream from the user, where the annotated media stream includes at least one item of supplemental content mapped to at least a portion of a media stream to produce the annotated media stream, storing the annotated media stream, receiving a request from a viewer to view the annotated media stream, and distributing the annotated media stream to the viewer. In another embodiment, a method for distributing an annotated media stream provided by a user includes distributing the annotated media stream to a viewer, where the annotated media stream includes at least one item of supplemental content mapped to at least a portion of a media stream to produce the annotated media stream, and collecting a commission when the annotated media stream is viewed by the viewer. In one embodiment, the collecting requires further action on a part of the viewer (such as selection of a hyperlink contained in the at least one item of supplemental content or consummation of a commercial transaction).

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefits of U.S. Provisional PatentApplication Ser. No. 60/788,367, filed Mar. 30, 2006; Ser. No.60/788,368, filed Mar. 30, 2006; Ser. No. 60/747,379, filed May 16,2006; Ser. No. 60/804,091, filed Jun. 6, 2006; Ser. No. 60/866,803,filed Nov. 21, 2006; and Ser. No. 60/871,705, filed Dec. 22, 2006, allof which are herein incorporated by reference in their entireties.

FIELD OF THE INVENTION

The present invention relates generally to advertising and relates moreparticularly to advertising relating to viewable media streams.

BACKGROUND OF THE INVENTION

With the increasing use by television viewers of digital video recorders(DVRs) to avoid traditional commercial advertising on television and insimilar media, product placement is increasingly becoming important as atool for advertisers to reach consumers. For example, an advertiser mayarrange for a particular article to be used in a movie by a popularactress, with the expectation that such use will project a positiveproduct image to consumers and thereby increase sales.

However, product placement is not without its drawbacks. For one,artistic concerns typically dictate that product placement besubstantially less obtrusive than traditional commercial advertising(e.g., prominent and repeated display of the name of the article isgenerally avoided), thus, it may not be clear to viewers what specificarticle or brand is being depicted. Additionally, even if the article isidentifiable, a viewer's potential interest in learning more about or inpurchasing the article may be eroded by the time delay between thearticle being depicted and the viewer having access to the informationor to a purchasing opportunity.

Several methods have been proposed for allowing consumers to interactwith media depicting product placements, for example to retrieveadditional information about a depicted article on demand. However, theenabling technology is still in its relatively early stages, and theuser interfaces are therefore typically cumbersome. Moreover, theimplementation methodologies tend to be computationally complex andtherefore quite expensive. In addition, if the advertising informationrelating to a product placement is embedded in the media stream itself,it is generally impractical or impossible to add, modify or delete thisinformation. This decreases the value of the advertising over time, dueto changes in various characteristics of the placed article, such as thearticle's retail price and/or availability.

Moreover, these same concerns often carry over to other popular but lesstraditional forms of media. For instance, services that provideon-demand access to video content over the Internet or anotherdistribution channel (e.g., wherein a filmmaker creates and then uploadsa video to an access provider and the access provider then makes thevideo available for viewing by others, potentially including the generalpublic) are becoming increasingly popular. In some cases, the accessprovider provides additional functionality, such as enabling viewers tosearch for specific video content of interest, ranking the videos byviewer popularity, and allowing the community of viewers to rate andpost commentary about particular videos.

Providing such services is expensive, particularly with respect to thestorage costs and bandwidth transmission costs associated withpresenting significant volumes of video content to a large audience.Therefore, a robust revenue stream is required to make a video accessprovider financially viable and sustainable. Present access providersattempt to generate revenue by various means including by charging usersa video hosting fee, by charging viewers a subscription or pay-per-viewfee, or, most popularly, by presenting advertising information to theviewing audience via pre-video or post-video commercials, banner ads,pop-up ads, and other types of advertising. In the latter case, revenuemodels for online video access providers typically face the samechallenge that traditional media faces: how to associate effectiveadvertising with the video content while not appearing to intrude on theviewer experience or on the creative integrity of the filmmaker.

In naïve advertising models, the advertising that is presented toviewers of online video is selected randomly and without reference tothe subject matter or other characteristics of the specific videocontent being viewed. In more sophisticated models, access providersstrive to improve the relevancy of the advertising they present byleveraging the data and processes that are intended primarily to enableviewers to find video content of interest. This can be accomplished, forexample, by referencing information from the video itself (generated,for example, using speech recognition, text recognition, or othertechnologies for determining the subject matter of a given video) and/orby referencing the search terms that have been input by viewers in theirefforts to locate a specific video.

Although such methods have proven to be somewhat effective in enablingvideo “search” in the online video context, the subject matter-relatedinformation that is obtained and/or utilized by such methods isgenerally too inaccurate, too misleading, too vague, too specific,and/or too voluminous to enable the effective selection and presentationof relevant advertising.

The deficiencies described above are compounded by the fact thateffective advertising targeting requires subject matter information tobe comprehended within the context of the overall viewing experience.Currently, the automated systems that are necessary to deal with thetremendous volume of videos that are hosted by successful online videoaccess providers (particularly with respect to “user-generated” content)are incapable of addressing such contextual elements. For example, evenif an automated system were to recognize that a given video containedsubject matter regarding “buying a new sport utility vehicle (SUV),” thesystem might fail to comprehend and/or take into account the fact thatthe context of the subject matter was a warning about the adverse impactof SUVs on global warming (and, as such, represented an undesirableadvertising placement for an SUV manufacturer).

Furthermore, current advertising targeting systems lack the capacity toaddress the important connection between the subject matter and contextof a video and the demographics and interests of likely viewers of suchvideo. For example, absent specific instructions that have beenformulated in advance (and are, therefore, very costly to provide on alarge scale), current systems are incapable of surmising that a viewerwho is interested in viewing a video warning against purchasing an SUVmight also be interested in viewing advertising about seeminglyunrelated subject matter such as organic foods, energy-saving lightbulbs, trips to see the melting glaciers of Kilimanjaro or the campaignof a local “Green” politician.

Finally, many of the methods that are typically used by access providersto present advertising (e.g., pre-roll, pop-up, or overlay advertising)are intrusive and/or disruptive of the viewing experience, therebyinterfering with the filmmaker's creative control over the presentationof his or her video and marring the quality of the viewer's viewingexperience.

For these and other reasons, advertising methods conventionally used byonline video access providers cause filmmakers to be less willing tocreate or share videos, viewers to be less willing to watch videos, andadvertisers to place a lesser value on the advertising “inventory”offered for sale by the access providers. As a result, online videoaccess providers fail to maximize the potential revenue opportunityrepresented by their viewing audiences.

Thus, there is a need in the art for a method and apparatus forannotating media streams such as television, hosted online video and thelike to provide more effective, user-friendly advertising.

SUMMARY OF THE INVENTION

In one embodiment, the invention is a method and apparatus forannotating media streams. One embodiment of a method for facilitatingcreation of an annotated media stream by a user includes receiving themedia stream from the user, mapping at least one item of supplementalcontent to at least a portion of the media stream to produce theannotated media stream, and storing the annotated media stream.

In another embodiment, a method for distributing an annotated mediastream provided by a user includes receiving the annotated media streamfrom the user, where the annotated media stream includes at least oneitem of supplemental content mapped to at least a portion of a mediastream to produce the annotated media stream, storing the annotatedmedia stream, receiving a request from a viewer to view the annotatedmedia stream, and distributing the annotated media stream to the viewer.

In another embodiment, a method for distributing an annotated mediastream provided by a user includes distributing the annotated mediastream to a viewer, where the annotated media stream includes at leastone item of supplemental content mapped to at least a portion of a mediastream to produce the annotated media stream, and collecting acommission when the annotated media stream is viewed by the viewer. Inone embodiment, the collecting requires further action on a part of theviewer (such as selection of a hyperlink contained in the at least oneitem of supplemental content or consummation of a commercialtransaction).

BRIEF DESCRIPTION OF THE DRAWINGS

The teaching of the present invention can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 is a schematic diagram illustrating one embodiment of atelevision value chain, according to the present invention;

FIG. 2 is a flow diagram illustrating one embodiment of a method forproducing coded media streams;

FIG. 3 is a flow diagram illustrating one embodiment of a method fordetecting and readings tags on articles appearing in a media stream,according to the present invention;

FIG. 4 is a flow diagram illustrating another embodiment of a method forproducing coded media streams;

FIG. 5 is a flow diagram illustrating another embodiment of a method forproducing coded media streams;

FIG. 6 is a flow diagram illustrating one embodiment of a method forannotating media streams, according to the present invention;

FIG. 7 is a flow diagram illustrating another embodiment of a method forannotating media streams, according to the present invention;

FIG. 8 is a flow diagram illustrating one embodiment of a method forproviding annotated media streams to viewers, according to the presentinvention;

FIG. 9 is a flow diagram illustrating a second embodiment of a methodfor providing annotated media streams to viewers, according to thepresent invention;

FIG. 10 is a schematic diagram illustrating an exemplary display forpresenting retrieved information about articles appearing in a portionof a media stream;

FIG. 11 is a flow diagram illustrating a third embodiment of a methodfor providing annotated media streams to viewers, according to thepresent invention;

FIG. 12 is a flow diagram illustrating a fourth embodiment of a methodfor providing annotated media streams to viewers, according to thepresent invention;

FIG. 13 is a high level block diagram of the media stream annotationmethod that is implemented using a general purpose computing device;

FIG. 14 is a flow diagram illustrating another embodiment of a methodfor annotating media streams, according to the present invention;

FIG. 15 is a flow diagram illustrating a fifth embodiment of a methodfor providing annotated media streams to viewers, according to thepresent invention;

FIG. 16 is a flow diagram illustrating one embodiment of a method forassociating supplemental content with a media stream, according to thepresent invention;

FIG. 17 is a flow diagram illustrating one embodiment of a method forproviding a service for hosting a media stream with at least one mappingto supplemental content;

FIG. 18 is a flow diagram illustrating one embodiment of a method fordisplaying a media stream with mapped supplemental content to a viewer;

FIG. 19 depicts an exemplary display device;

FIG. 20 illustrates an exemplary media stream that has been logicallydivided into a plurality of segments;

FIG. 21 illustrates an exemplary mapping in which segment identifiersidentifying segments of the media stream of FIG. 20 are mapped to itemsof supplemental content;

FIG. 22 illustrates an exemplary mapping in which segments of the mediastream illustrated in FIG. 20 are associated with supplementalinformation;

FIG. 23 illustrates an exemplary filming location in which a pluralityof articles are tagged;

FIG. 24 depicts an image capturing device with integrated RFID sensorsystem;

FIG. 25, for example, illustrates an exemplary tag log that identifiesthe active tags associated with each segment identifier identifying amedia stream segment;

FIG. 26 illustrates an exemplary set-top box that interacts with usersignals provided via a remote control;

FIG. 27 illustrates the system of FIG. 26 after the supplemental contentmapped to the segment identifier has been retrieved and displayed to theviewer via the display device;

FIG. 28 illustrates the system of FIG. 26 after the viewer has indicatedinterest in a single article for which supplemental content is mapped;

FIG. 29 illustrates the use of viewer-driven constraints in the displayof supplemental content;

FIG. 30 illustrates the results of using viewer-driven constraints toselect supplemental content;

FIG. 31 illustrates an exemplary system in which a viewer signalindicates that the viewer wishes to “bookmark” supplemental contentmapped to the current segment's segment identifier for later viewing;

FIG. 32 illustrates an exemplary media stream that has been loaded ontoa computing device;

FIG. 33 illustrates an exemplary item of supplemental content that maybe mapped to the media stream illustrated in FIG. 32;

FIG. 34 illustrates the exemplary media stream of FIG. 32, where themedia stream has been logically divided into temporal segments tofacilitate mapping of supplemental content;

FIG. 35 illustrates the exemplary media stream of FIG. 32, wherein anitem of supplemental content is mapped such that the item ofsupplemental content is displayed simultaneously with at least a segmentof the media stream;

FIG. 36 illustrates three items of similar supplemental content forpotential mapping to a segment identifier associated with a segment of amedia stream;

FIG. 37 illustrates click-through counters for each respective item ofrotated supplemental content illustrated in FIG. 36;

FIG. 38 illustrates click-through counters for different respectiveitems of rotated supplemental content;

FIG. 39 illustrates the dynamics of an exemplary syndication team,according to the present invention;

FIG. 40 illustrates an exemplary initial display for presentation to acontent creator;

FIG. 41 illustrates an exemplary display that may be presented to anadvertiser to allow search of media streams associated with a given itemof supplemental content;

FIG. 42 illustrates an exemplary display in which results of a searchsuch as that enabled by the display of FIG. 41 are presented;

FIG. 43 illustrates an exemplary display in which an advertiser isenabled to block association of an item of supplemental content with agiven media stream;

FIG. 44 illustrates an exemplary scenario in which the ability toprovide an item of supplemental content relating to a given articleappearing a media stream is auctioned off to various sellers;

FIG. 45 illustrates an exemplary display in which the supplementalcontent associated with a media stream is used to generate relatedbanner advertising;

FIG. 46 illustrates the use of an initial annotated media stream acrossvarious platforms including a social networking website, a web log and aportable media player;

FIG. 47 illustrates another application of aspects of the presentinvention; and

FIG. 48 illustrates another application of aspects of the presentinvention.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

DETAILED DESCRIPTION

In one embodiment, the present invention relates to the annotation ofmedia streams, including video signals. Embodiments of the inventionenable users (e.g., advertisers, content creators and the like) toannotate media streams such as television shows, commercials,infomercials, trailers, shopping channel segments, music videos, movies,VHS or DVD recordings, video-on-demand content, video games (e.g.,interactive television games, personal computer games, online games,handheld games and console-based games), “podcasts”, webcasts (e.g.,teleconferences, educational broadcasts, etc.), hosted online video andthe like with a mapping to supplemental content relating to, forexample, advertising copy for articles that appear in or are otherwiserelated to the media streams (e.g., commercial articles, actors, songs,etc.). Thus, product placement and the presentation of relatedadvertising may be implemented in a manner that is computationallysimplified from a technical perspective and substantially unobtrusivefrom a viewer perspective. Moreover, embodiments of the invention enablethe generation of a robust revenue stream that may, in turn, providesupport for infrastructure and encourage use by content creators,advertisers and viewers. In addition, media streams may be annotated andmapped to items of supplemental content of a non-commercial nature, suchas content that is merely informative or aesthetic.

In one embodiment, the annotated media stream is a viewable mediastream. By “viewable”, it is meant that the media stream comprises atleast a video signal, and may additionally comprise other components,such as an audio signal or a text signal. In an alternative embodiment,the media stream does not have a viewable component, but comprises atleast an audio component (e.g., streaming music, radio stationbroadcasts, audio feeds stripped from video streams) or a text component(e.g., streaming text, rich site summary, really simple syndication(RSS) or real-time simple syndication feeds, internet relay chatchannels). For example, in one embodiment, the media stream comprises avideo signal for display on a display device such as at least one of: atelevision, a personal computer, a mobile telephone, a personal digitalassistant, a portable media player, an in-flight or in-vehicle systemdisplay, or a “head-up” display. Where the media stream comprises anaudio signal, the user device may comprise at least one of: earphones,headphones or speakers.

In a specific embodiment of the invention, methods are provided forannotating media streams (such as video content) uploaded to an onlinevideo distribution and access provider with mappings to supplementalcontent. The specific content and presentation of the supplementalcontent, including advertising and viewer interactive opportunities, maybe dictated by the user who uploads the media stream (e.g., the contentcreator), or this user may permit collaborators to select and associatethe supplemental content. By enabling the uploading user to control theselection of supplemental content, the invention enables him/her toleverage his/her contextual understanding of the media stream contentand of what items of supplemental content may most appeal to likelyviewers of the media stream. As a result, more relevant and, thus, moreeffective advertising is presented to viewers. Further, as a componentof a business model, the invention allows the uploading user (andpotentially collaborators) to monetize the media stream content.

Thus, unlike current methods for presenting advertising in the onlinevideo context, embodiments of the present invention leverage the factthat presenting media streams on the Internet creates a unique potentialfor viewers to interact with associated advertising (e.g., viewer“click-throughs”) and for that interaction to be monitored, analyzed,and acted upon. Also leveraged is the fact that, unlike conventionaltelevision distribution, hosted media and video-on-demand is presentedand viewed at times selected by each individual viewer, and notsimultaneously by a large number of viewers according to a schedule.Thus, as viewing events for the same media stream in a hosted mediacontext are distributed across a range of time, viewer interactions maybe monitored and used to enhance the experience of subsequent viewers.Embodiments of the present invention take advantage of this opportunityto optimize the presentation of advertising, for example by showingdifferent selections of advertising to the early viewers of a mediastream (e.g., the first 1,000 viewers), monitoring viewer response toeach of those advertisement selections, and then using that informationto identify and present the most compelling advertising to allsubsequent viewers of the media stream and/or to those early viewerswhen they view this, or other, media streams in the future.

FIG. 1 is a schematic diagram illustrating one embodiment of a broadcastvalue chain 100, according to the present invention. Specifically, thevalue chain 100 depicts a television value chain, but those skilled inthe art will appreciate that the general concepts embodied therein maybe extended to application substantially in any form of mediadistribution. The value chain 100 comprises a number of entities,including at least: one or more content creators 102, one or moreviewers 104, one or more access providers 106, one or more advertisementbuyers 108, one or more sellers 110 and a stream annotation serviceprovider 112.

Content creators 102 comprise studios, independent production companiesand even individuals who create media streams for distribution toviewers, such as television shows, movies, commercials and videosavailable over the Internet. The content creators 102 provide the mediastreams to one or more access providers 106, such as cable, satellite,Internet and telecommunications service providers or World Wide Webhosting sites, who in turn provide the media streams to the viewers 104who receive the media streams (e.g., via televisions, personalcomputers, personal digital assistants or other display devices). Inaccordance with the present invention, the viewers 104 may interact withthese media streams via graphical user interfaces operated inconjunction with display devices, remote controls, set top boxes and/orother input/output (I/O) devices.

Sellers 110, such as retailers, manufacturers, and service providers,produce products and services for selling to consumers (e.g., viewers104). As described above, a common way of advertising such products andservices is to purchase advertising time in media streams (e.g., in theform of traditional commercials that interrupt the main program contentor in the strategic placement of products within the main programcontent) from content creators 102. Often the purchase of advertisingtime or product placement is brokered by an advertisement buyer 108,such as a product placement agent or an advertising agency, who tracksnew media releases, purchases advertising time or placements directlyfrom the content creators 102 and places advertisements or products onbehalf of the sellers 110. As described above, this brokering often mustbalance competing concerns, such as the sellers' desire to have theirarticles placed in a recognizable manner and the content creators'desire to maintain the artistic integrity of the content, not to mentionthe viewers' desire not to be inundated or bothered by overly intrusiveadvertising content. Conventionally, with respect to purchasers ofadvertising time, the seller 110 or its agent then provides theadvertising content, possibly brokered by the advertisement buyer 108,to the access provider 106, who then inserts the advertising content inthe media stream for distribution to the viewers 104.

According to the present invention, a stream annotation service provider112 is an intermediary who facilitates the balance of competing concernsby enhancing the efficacy of traditional product placement. As will bedescribed in further detail below, the stream annotation serviceprovider 112 works with the content creators 102, the sellers 110 and/orthe access providers 106 to create substantially unobtrusive advertisingfor media streams that are distributed to viewers 104. In oneembodiment, the stream annotation service provider 112 receivesspecially coded media streams from the content creators 102. Inparticular, the media streams are coded into a plurality of segmentsidentified by (possibly unique) segment identifiers, an index of whichis provided to the stream annotation service provider 112. The index mayspecify, by segment identifier, which particular segments of the mediastream depict particular articles or types of articles that may besuitable for association with advertising content. In addition, thestream annotation service provider 112 receives advertising content fromthe sellers 110. In accordance with the present invention, the streamannotation service provider 112 then maps the advertising content tosegment identifiers associated with segments of the media stream (e.g.,in accordance with the index) and stores this mapping. The mapping isthen provided to the access providers 106, who can then provide theadvertising content to the viewers 104, in accordance with the mappingand the segments of the media stream being viewed.

For example, suppose a content provider 102 has created a media streamcomprising a television show, and, at a given point in the televisionshow, an actress is depicted wearing a particular sweater. A retailer(e.g., a seller 110) who sells the sweater could, in accordance with thepresent invention, contact the stream annotation service provider 112for assistance in advertising the commercial availability of thesweater. In one embodiment, the stream annotation service provider 112maps advertising content relating to the sweater to one or more segmentidentifiers associated with the corresponding segments of the mediastream, such that when the media stream is distributed to viewers 104, aviewer 104 can access the advertising content on-demand by sending asignal to the access provider 106 at an appropriate time. Theadvertising content could include commercial information about thesweater (e.g., sellers from whom it is commercially available,materials, colors, etc.), as well as an option to immediately purchasethe sweater (e.g., via the seller's Internet web site, or by interactivetelevision shopping facilities).

In further embodiments described in greater detail below, the streamannotation service provider 112 may also perform data mining tasks inaddition to facilitating effective product placement advertising. Forinstance, the access provider 106 and/or stream annotation serviceprovider 112 may log received viewer signals as a means of gauging theinterests of individuals or groups of viewers 104 (e.g., by analyzingthe types of products and services for which the viewers 104 requestadvertising). Such information can be provided to sellers 110 and/oradvertisement buyers 108 to aid them in tailoring the advertisementsmade available to certain individuals or groups of viewers 104, therebyallowing sellers to maximize the efficacy of their product placements.In addition, such information can be useful in assessing the efficacy ofparticular product placements and in determining appropriate pricingmodels for these placements.

Although the above describes a particular arrangement in which discreteentities perform certain tasks or functions, those skilled in the artwill appreciate that in further embodiments, some entities could providemultiple functionalities. For instance, in an alternative embodiment,stream annotation functionality, as provided by the stream annotationservice provider 112 above, could be provided by another party, such asby an access provider 106 or a content creator 102. Alternatively, thestream annotation service provider 112, the access provider 106 or thead buyer 108 may be responsible for indexing the segment identifiers ofthe media stream for mapping to advertising content. In furtherembodiments still, the content creator may be responsible for not onlycreating the media stream, but for creating advertising content andmappings. Thus, the present invention contemplates many differentarrangements for carrying out the same tasks. Stream annotation servicesaccording to the present invention thus leverage the existing televisionvalue chain.

It may not be cost effective for the content creators to invest muchtime in pursuing the sellers of the many “minor” products depicted in agiven media stream (e.g., articles that appear for only a short time,articles that aren't amenable to generating a viewer response, etc.).Put another way, it is relatively easy and inexpensive for the contentcreator to simply record the name and seller of each article in aproperty list; however, it is relatively time intensive to go out andtry to “sell” placement of an article.

In one embodiment, the stream annotation service provider 112 hosts awebsite that lists articles (e.g., products, services, music, people) ortypes of articles that appear in a given media stream or set of mediastreams (e.g., a bronze lamp from the 1960s). Sellers may view thewebsite and search for articles of interest, potentially as sucharticles have been placed within a library or ontology of articles, asdiscussed in greater detail below. For example, a lamp seller may wishto provide supplemental content (e.g., advertising content) forpresentation to viewers featuring an article that is similar to anarticle actually depicted in a media stream, but that may not be theexact depicted article. If the seller is sufficiently interested, theseller may immediately pay a fixed fee cost for mapping his/hersupplemental content for display to viewers (e.g., a picture of thearticle, more information etc.). The revenue derived from this servicemay be shared with the content creator who created the media stream,with actors featured in the media stream and the like.

In one embodiment, the stream annotation service provider 112, insteadof acting as the market enabler for media stream annotation (i.e.,providing the technical infrastructure), the market maker (i.e., theservice division brokering placement deals), or the clearing house forthe entire media stream market, the stream annotation service provider112 could choose to be an active participant in the market. That is, thestream annotation service provider 112 could purchase the right (for,say, a flat fee) to associate supplemental content with respect to aparticular media stream, and then sell this right (in whole or in parts)to the highest bidder after adding the value of clickstream targetinginformation. Utilizing this approach, the content creator is providedwith a predictable price for its content, the stream annotation serviceprovider 112 bears the risk (and the resulting profit potential) ofmaking the resale, and the sellers benefit because the stream annotationservice provider 112 provides a single point of coordination.

FIG. 2 is a flow diagram illustrating one embodiment of a method 200 forproducing coded (i.e., indexed) media streams. The method 200 may beimplemented, for example, by a content creator 102 such as thoseillustrated in FIG. 1, in order to facilitate media stream annotation bya stream annotation service provider such as the stream annotationservice provider 112. Alternatively, the method 200 could be implementedentirely at the stream annotation service provider 112.

The method 200 is initialized at step 202 and proceeds to step 204,where the method 200 identifies one or more articles to appear in themedia stream (i.e., prior to filming of the media stream) and tags eachof the identified articles with a (possibly unique) tag. For example,FIG. 23 illustrates an exemplary filming location 2300 in which aplurality of articles are tagged (i.e., “RFID”).

In one embodiment, every article to appear in the media stream isidentified and tagged. In one embodiment, the articles are tagged withrandom tags. In another embodiment, one or more particular articles areallocated specific, unique tags for use in all media streams (e.g., softdrink X will always be tagged with tag Y, regardless of the mediastream(s) in which soft drink X is being used and/or regardless of thecontent creator filming the media stream). In this embodiment, auniversal database accessible by the content creator might specify theappropriate tag to be used for a given article. Such a database could bebuilt and maintained by, for example, the stream annotation serviceprovider 112. In one embodiment, the identified articles are tagged withunique radio frequency identification (RFID) tags.

In step 206, the method 200 creates or updates a database of tags andassociated articles (e.g., database 2302 of FIG. 23). In one embodiment,the database entry for an article includes, in addition to the ID of theassociated tag, information about the article itself (e.g., as suppliedby the seller 110 of the item). In another embodiment, the databaseentry for an article includes at least one of: a description of thearticle, an image of the article or a hyperlink or other identifierindicating that the article is already registered with the streamannotation service provider 112 (in which case, information about thearticle could be pre-loaded in accordance with the seller's wishes). Ina further embodiment, the database entry for an article includes ameasure of compensation that a seller of the article is willing toprovide in exchange for placement and advertising of the article in amedia stream.

In step 208, the method detects and reads the tags on tagged articles,in substantially real time as the media stream is filmed. In oneembodiment, tags are detected and read by a sensor system that isdeployed at the site at which the filming of the media stream occurs(and deployed at an appropriate read distance from the set including thetagged articles). In an alternative embodiment, the sensor system isbuilt into the set (e.g., into the ceiling, the floor and/or themicrophone system), rather than linked to the image capturing device(s),to minimize read distance issues. The latter embodiment would beefficient and cost effective particularly for filming locations that arere-used (i.e., not mobile). In another embodiment, the sensor system isa handheld sensor system operable by an individual who walks around thefilming location either before or after filming and scans the taggedarticles. In this case, information recorded by the handheld sensorsystem is sent (e.g., via a wireless transmitter or via a network towhich the sensor system may be connected) to a decoder for furtherprocessing. In another embodiment still, the sensor system is integratedinto an image capturing device's field of view or is built intoarticulating arms that are calibrated and programmed to move incoordination with the image capturing device(s), such that only tags onarticles appearing in the image capturing device's field of view (i.e.,what is actually filmed as being “in the shot”) are read. Thisembodiment could be achieved in conjunction with a triangulation systemto exclude tags outside of the image capturing device's field of view.In the embodiment where the tags are RFID tags, the sensor system is anRFID sensor system that is capable of interrogating the RFID tags.

For example, as illustrated in FIG. 24, which depicts an image capturingdevice 2400 with integrated RFID sensor system 2402, the field offilming defined by the image capturing device's field of view and thefield of tag activation defined by the sensor system's range highlightthe same area of the filming location 2406. Thus, tags that areactivated by the sensor system 2402 are read as the tagged articlesappear in the image capturing device's field of view.

In step 210, the method assigns (possibly unique) identifiers to eachframe of the media stream, in substantially real time as the mediastream is filmed. In one embodiment, identifiers are assigned to filmedsegments (e.g., individual frames) in accordance with an encoder that iscommunicatively linked to the image capturing device(s) (e.g., videocamera, still camera or the like) that is filming the media stream. FIG.20, for example, illustrates an exemplary media stream 2000 that hasbeen logically divided into a plurality of segments 2002 ₁-2002 _(n)(hereinafter collectively referred to as “segments 2002”). In theillustrated example, each segment 2002 comprises one frame of the mediastream 2000. Moreover, as illustrated, each segment 2002 is assigned aunique segment identifier—for example, the segment identifieridentifying segment 2002 ₁ is “XYZ-100”; the segment identifieridentifying segment 2002 ₂ is “XYZ-101”; the segment identifieridentifying segment 2002 ₃ is “XYZ-102”; and so on. The “XYZ” portion ofthe segment identifier is a program identifier identifying the mediastream 2000, and the numerical portion of the segment identifier is anidentifier identifying the specific segment in the media stream 2000.

It is to be noted that the identifiers assigned to the segments in step210 may not be the “final” segment identifier that will be transmittedwith the completed (and potentially edited) media stream, but theassigned identifiers could be used to generate the final segmentidentifiers, such as when the streams generated by multiple differentimage capturing devices that have filmed the same scene are editedtogether into a single, integrated media stream. Put another way,later-stage encoding may embed the identifiers assigned in step 210 inmetadata or the like, as described in greater detail below.

In one embodiment, the segment identifier is embedded as standard “text”in a closed captioning stream associated with the media stream, but withthe text altered so that it is invisible to viewers of the media streamand closed captioning stream (e.g., the text of the embedded segmentidentifiers is presented in the same color as the background or istransparent). This approach would leverage existing systems for addingembedded information to media streams.

In one embodiment, an encoder “reads” the media stream it is encoding inorder to obtain the program identifier (e.g., the International StandardAudiovisual Number or ISAN) that is embedded in the media stream, andthen performs one or more of the following operations: (a) generating asegment identifier for each segment (possibly unique as compared to theidentifiers of the other segments in the media stream); (b) combiningthe program identifier with the segment identifier, potentially precededby a unique signifier that identifies that the program and segmentidentifiers are to be used by a relevant processor (e.g., a set-top box,an access provider's system) to differentiate the program and segmentidentifiers from other information that may be embedded in the mediastream; and (c) automatically embedding the entire code in whateverform(s) are required, such as in the “invisible” text of the closedcaptioning, the “metadata” section of an Internet Protocol Television(IPTV) frame, the vertical blanking interval, a watermark or the like.

Different types of media streams have different properties (e.g.,different frames-per-second rates), and when one media stream that hasbeen encoded with segment identifiers in a first format is translatedinto a second format, it may be necessary for the segment identifiercoding to also change so as to be consistent with the second format. Inone embodiment, an encoder reviews the coding embedded in a “master”media stream, references the appropriate adjustment factors (e.g., theadjustments necessary to account for the differing frame rates or othercharacteristics), and then encodes a “subordinate” media stream with theadjusted coding. This process may be undertaken simultaneously with morethan one subordinate media stream.

Referring back to FIG. 2, once the tagged articles have been read andthe segments of the media stream have been assigned segment identifiers,the method 200 proceeds to step 212 and creates an index of articles tosegment identifiers, in accordance with the data gathered and/orassigned in steps 208 and 210. In the embodiment where an encoder isused to assign identifiers to segments of the media stream, the encodermay be communicatively linked to the sensor system so that, as eachsegment of the media stream is created during filming, the RFID codes onthe RFID tags are read and simultaneously recorded. The index may thenbe stored in a database (e.g., resident in the encoder or at a centralcollection point linked via a network).

The method 200 terminates in step 214.

The method 200 thereby enables the automated, real-time indexing ofarticles appearing in a media stream to segment identifiers of the mediastream in which the articles appear. These indices are then provided tothe stream annotation service provider 112 to facilitate markup of themedia stream with metadata related to the articles appearing therein, asdescribed in greater detail below. It should be noted that while such anindex could also be performed by an individual armed with a propertylist of articles appearing in the video, who watches the media streamand records frames in which the articles appear, the method 200 greatlysimplifies the indexing process. For example, the method 200substantially reduces the number of people and the amount of timerequired to produce the index, thereby lowering cost and speedingproduction. In one embodiment, where scenes (i.e., ranges of frames) arecut from the final version of the media stream, the raw encoding orindex may be replaced with a final index that is ultimately provided tothe stream annotation service provider 112. The frames of this finalindex may be associated with both their raw identifiers and their finalidentifiers.

In one embodiment, segments of the media stream (i.e., sequences offrames) are identified (e.g., in the stored index) by (possibly unique)segment identifiers. In one embodiment, the segments identifierscomprise sets of frame numbers associated with the segments (e.g.,frames a through c). In another embodiment, the segment identifierscomprise a generated number (e.g., generated by hashing, and thenoptionally encrypting, frames of the segment). For example, in oneembodiment, the segments are temporal segments identified by timecodesor sets of timecodes (e.g., t through t+3) associated with one or moreframes of the media stream that occur during the temporal segment. Insome embodiments, Moving Picture Experts Group (MPEG) or Society ofMotion Picture Television Engineers (SMPTE) time-codes may be used assegment identifiers. In one embodiment, the segment identifiers areassociated with segments in a manner such that, when a media streamcomprising segments is edited or otherwise transformed, the segmentidentifiers continue to be associated with the same portion of theprogram content (e.g., by embedding the segment identifiers in the mediastream such that the segment identifiers are directly associated withindividual frames).

In an alternative embodiment, the segments correspond to observed timeswithin the media stream (e.g., similar to, for example, stopwatchreadings where a stopwatch is started at the same time as the mediastream), thereby comprising timing information relative to a knownposition (e.g., the start of the media stream or the start of a scenewithin the media stream). Alternatively, the number of frames within themedia stream from a known position is used. Preferably, the offset is apositive offset (i.e., relates to a point in the media stream thatoccurs after the known position), although it is contemplated that therelative position may also be negative.

In a further embodiment still, the media stream may be divided into arelatively small number of first-level segments (e.g., divide atelevision show into segments corresponding to entire scenes or portionsoccurring between commercial breaks or a chapter of a DVD), each ofwhich is further associated with a series of second-level sub-segments,comprising time identifications (e.g., stopwatch readings) relative tothe start of the first-level segment. For example, the media streamcould be divided into n first-level segments, S₁, S₂, . . . , S_(n).Each of these first-level segments is then further associated withrelative timing information that corresponds to particular sub-segmentsof the first-level segment, according to the embodiments describedearlier herein, just as if the first-level segment were the completemedia stream. This would improve the accuracy of the association impliedby the index, given that the times associated with each segment startfrom a fixed point in a (presumably) uninterrupted stream.

In further embodiments of the method 200, entries in the index (i.e.,entries for articles and corresponding frames) are further augmentedwith scene information (e.g., actors appearing in the scene, location ofthe scene, etc.). This additional annotation could aid sellers andadvertisement buyers in understanding the contexts in which theirarticles are to be presented. Alternatively, this augmentation could beperformed by the stream annotation service provider 112.

In a further embodiment still, an index entry identifies a frame orseries of frames that depict an indexed article to its greatestadvantage. For example, the frame or frames may depict a “best” view ofthe article from a seller's (or buyer's) standpoint, such as the segmentin which the entire article is visible for an extended period of time.In this embodiment, the article may be associated with multipledifferent “scenes”, such as a “full” or theatrical scene that depictsthe article in an advantageous context (e.g., a particular article isfeatures in a crowd-pleasing scene involving two popular actors), and an“advertising” scene that depicts a few seconds before, during and afterthe article appears. In one embodiment, a user may be enabled to selecta scene from the index entry based on specified parameters (e.g., “allshoes worn by the lead singer in any music video featuring the bandXYZ”).

In one embodiment, a database of tags and articles is maintained inorder to identify articles that are mapped to different tags but thatare likely the same article. For example, one content creator mightidentify a particular pair of sunglasses as the “Brand X Model Y, Men'sSunglasses” whereas another content creator might use the phrase “ModelY Sunglasses by Brand X.” By identifying such potential matches, a usercould easily review such possible connection, change the descriptions sothat the descriptions are more uniform, and/or formalize a link betweenthe two articles so that tasks such as updating related advertisingoutput can be done more easily and accurately. Thissearch/identification process would potentially reach out into databasesmaintained by different content creators and/or by an access provider orstream annotation service provider.

In one embodiment, a user is not permitted to modify data from anoutside database (e.g., a third-party provided catalog of articles). Theuser is granted read access only, but is provided with a mechanism toeasily notify the relevant parties of the potential link between twoarticles and to administer parallel changes. For example, a reviseddescription that both parties can download could be provided by a thirdparty such as an access provider or stream annotation service provider.Alternatively, a collaborative tool may be provided in which bothparties (and perhaps third parties such as advertisement brokers orsellers) could offer their suggestions before finalizing the newdescription.

In one embodiment, one or more RFID tags are recorded by the contentcreator to include information regarding a particular filming situation.For example, an entire segment might comprise an action sequenceinvolving martial arts, and the film crew (e.g., at the direction oftheir marketing department) might want to have that informationrecorded. For example, knowing that the scene involves martial artsmight enable certain marketing information/metrics to be generated whena viewer ultimately signals an interest in obtaining supplementalcontent during presentation of the segment, as discussed in furtherdetail below (e.g., it could be learned that a particular viewer orclass of viewers responds well to video game advertising that followsaction sequences). Other examples might be to record a segment as a“musical sequence,” as something involving Venice Beach (e.g., aparticular location), as employing a certain type of filming technique(e.g., black and white only) or the like. All of these types ofinformation are independent of any given article in the segment but maynonetheless be of significant interest for the purposes of targetingsupplemental content.

In such circumstances, it will be useful to supplement the RFID taggingprocess discussed previously with RFID indexing that isarticle-independent, but that is still provided “live on set.”Accordingly, the “segment RFID tag” could be placed in close proximityto each RFID sensor located on an image capturing device, so that thesegment RFID tag is always indexed during filming, even though thesegment RFID tag is not in the field of view of the image capturingdevice. For example, the segment RFID tag could be inserted in a holdingbracket that is above the image capturing device lens but that is withinrange of the RFID sensor beam. In one embodiment, instead of using anRFID tag, an input device such as a keyboard could be used to inputsegment specific, item independent information for indexing in the filmalong with the RFID-sensed articles.

In one embodiment, if certain articles are physically too small in sizeto accommodate an RFID tag (or, indeed, there may be no article at allbut merely a “concept”), an RFID tag could be placed in proximity withthe actual article so that the film is indexed properly. For example, awrist watch might be too small to accommodate an RFID tag, but the shirtsleeve appearing right next to the watch would provide sufficientproximity to ensure accurate indexing.

FIG. 3 is a flow diagram illustrating one embodiment of a method 300 fordetecting and readings tags on articles appearing in a media stream,according to the present invention. In particular, the method 300provides a means of detecting and locating tagged articles in athree-dimensional space (e.g., a film set), via use of a sensor array asdiscussed above. The method 300 may be implemented, for example, inaccordance with step 208 of the method 200.

The method 300 is initialized at step 302 and proceeds to step 304,where the method 300 reads one or more article tags from one or morelocations in the three dimensional space (e.g., defined by sensorsplaced in the ceiling and/or floor, or in other locations such as amicrophone boom).

In step 306, the method 300 determines the three-dimensional (i.e., x,y, z) position of one or more of the articles, in accordance with thereadings. In one embodiment, the determination of an article'sthree-dimensional position is assisted by readings reflecting therelative signal strength of the article's tag. For example, if a sensorin the ceiling is positioned ten feet from the floor and it is known(e.g., by ultrasound, laser or other means) that a signal strength of sindicates a distance of four feet from the sensor, the “height” of anarticle whose tag emits a signal of strength s in the three dimensionalspace can be calculated as approximately six feet from the floor.

In step 308, the method 300 identifies a subset of the three-dimensionalspace detected by the sensor array that is defined by the range of viewof an image capturing device. In one embodiment, this subset space isdetermined by a framing process that first locates the image capturingdevice within the three-dimensional space (e.g., by physically measuringthe location of the image capturing device or by tagging the imagecapturing device itself with a tag that is readable by the sensorarray). This places the image capturing device relative to the scenebeing filmed for the media stream. Once the image capturing device'srelative position is known, readings are obtained from the imagecapturing device to determine the three-dimensional direction in whichthe lens of the image capturing device is pointed along with thezoom/field of view of the image capturing device (e.g., a panoramic shotversus a “tight” shot). These readings are then combined to produce thethree-dimensional subset space. This subset space is continuouslyrecalculated as filming takes place, since it will change as the imagecapturing device pans, tilts and zooms.

In step 310, the method 300 selects a tagged article in thethree-dimensional space defined by the sensor array. The method 300 thenproceeds to step 312 and determines whether the selected article appearsin the subset space determined by the framing process (i.e., step 308).This may be done by comparing the output of step 306 to the output ofstep 308.

If the method 300 concludes in step 310 that the selected articleappears in the subset space, the method 300 proceeds to step 314 andmarks the selected article as “active” in a tag log. FIG. 25, forexample, illustrates an exemplary tag log 2500 that identifies theactive tags associated with each segment identifier identifying a mediastream segment.

Alternatively, if the method 300 concludes in step 310 that the selectedarticle does not appear in the subset space, the method 300 proceeds tostep 316 and marks the selected article as “inactive”. In oneembodiment, a first article that is associated with a second article(e.g., a pair of eyeglasses worn by an actor) may be marked as activewithin the subset space whenever the second article is marked as active,even if the first article is marked as inactive within the subset spaceor is not marked at all.

In step 318, the method 300 determines whether to select another taggedarticle in the three-dimensional space defined by the sensor array. Inone embodiment, the method 300 selects another tagged article if thereis at least one tagged article that has not yet been located and markedas active or inactive. If the method 300 concludes in step 318 thatanother tagged article should be selected, the method 300 returns tostep 310 and proceeds as described above to select another tagged (butnot yet marked) article. Alternatively, if the method 300 concludes instep 318 that another tagged article should not be selected, the method300 terminates in step 320. Thus, the method 300 repeats or iteratesuntil all tagged articles that are meant to be marked (as active orinactive) are marked.

As described above with respect to FIG. 2, a substantially simultaneousprocess during filming could encode the frames of the media stream with(possibly unique) segment identifiers. A log file could match thesesegment identifiers with a list of articles that are “active” at themoments in time that the frames were created. In one embodiment, thislog file is organized article-by-article such that each article islisted only once, but is associated with all of the relevant segmentidentifiers of the media stream.

In one embodiment, the method 300 could be adapted to encode a livecontent feed. In such a case, the raw segment identifiers would be theonly identifiers associated with the segments of the media stream. Insuch a case, a live operator could supplement the index with additionalarticles that should be associated with the segment identifiers but thataren't marked as “active” (e.g., articles that were not tagged or thatare associated with “concepts” suggested by the media stream, such as abeach vacation, rather than with tangible, visible goods). In addition,such non-marked articles could be associated with an edited index (i.e.,as opposed to a raw index created for a live content feed). In addition,the operator may be enabled to edit the active/inactive designations ofarticles marked by the method 300 (e.g., to correct any inaccuracies).For example, the operator might access the set of “inactive” articles inorder to add articles to the index that were perceived by the automatedmethod 300 to be just out of a scene, but did, in fact, appear.

FIG. 4 is a flow diagram illustrating another embodiment of a method 400for producing coded media streams. Like the method 200, the method 400may be implemented by a content creator 102 in order to facilitatestream annotation by the stream annotation service provider 112.Alternatively, the method 400 could be implemented entirely at thestream annotation service provider 112.

The method 400 is initialized at step 402 and proceeds to step 404,where the method 400 creates or obtains an “encyclopedia” (i.e., adatabase) of images of one or more articles and/or individuals (e.g.,actors) that will potentially appear in a media stream. In oneembodiment, the images in the database are optimized for videorecognition purposes. This might include, for example, filming anarticle or individual from multiple angles or with different lighting tofacilitate later recognition (so that the article may be recognizedregardless of how the article is actually filmed, such as in daytimeversus nighttime scenes). In one embodiment, a database entry for anarticle or individual includes, in addition to one or more images, oneor more of the following: textual identifying information about thearticle or individual, an identification number that uniquely identifiesthe article or individual, an indication of whether the article orindividual is likely to be moving in the media stream (e.g., a car) orstatic and other data characterizing certain attributes that may enhancethe accuracy of a video recognition system (e.g., luminance, hue, orreflectivity). In another embodiment, the filming location is includedas an entry in the database (e.g., an outdoor scene featuring snow andice versus an indoor scene featuring the inside of a tent), enablingdifferentiation of foreground objects from the background scene. Inanother embodiment, a database entry for an article or individual couldbe selected from entries in another encyclopedia.

In step 406, the method 400 obtains a segment identifier-encoded mediastream (i.e., a media stream whose segments have already been encodedwith identifying information). The method 400 then proceeds to step 408and processes the media stream in accordance with a video recognitionsystem, using the encyclopedia of images created in step 404 as an inputto facilitate recognition. The relatively small size of the encyclopediashould enable more accurate recognition of articles appearing in themedia stream.

In step 410, the method uses the recognition results to associate one ormore segment identifiers of the media stream with one or more article(or individual) IDs for articles (or individuals) appearing in thecorresponding segment(s). In one embodiment, association of segmentidentifiers and article (or individual) IDs is contingent upon humanverification of the recognition results. Thus, an index of articles (andindividuals) to segments is created. This index may then be provided,for example, to the stream annotation service provider 112 for furtheruse and processing, as described in greater detail below. The method 400then terminates in step 412.

In further embodiments of the method 400, the video recognitionprocessing that is performed on the media stream (i.e., as in step 408)is performed simultaneously with the filming of the media stream (asopposed to being performed on a previously filmed media stream). Ifsegment identifiers are unique, the accuracy of the resultant indexshould be maintained regardless of any subsequent editing.

In a further embodiment still, the tagging techniques of the method 200could be combined with the method 400 for enhanced recognition results.For example, a tag ID reading could be used to cross-verify videorecognition processing results.

In one embodiment, a video recognition system is programmed to “learn”that certain combinations of articles tend to move together (e.g.,eyeglasses and earrings are likely to be in the same field of view andto be positioned near each other).

In another embodiment, a number representing the “zoom” of the mediastream is encoded therein (e.g., using the image capturing device(s)that filmed the media stream). This might be useful in terms of helpingdetermine the true size of shapes depicted within a segment of the mediastream. For example, knowing that a given segment of the media streamdepicts a “tight” shot and that both an earring and a necklace aredepicted in the segment might enable an automated recognition system toaccurately differentiate the earring from the necklace.

In one embodiment, the image capturing device(s) filming the mediastream is linked with a segment identifier encoder and with a videorecognition system (which is in turn linked to an encyclopedia ofarticles appearing in the media stream). In this case, both the segmentidentifier encoding and the mapping to supplemental content areperformed simultaneously with the filming of the media stream. If allsegment identifiers are selected so as to be unique, no matter how themedia stream is ultimately edited, the segment identifiers will stillrefer to the same items of supplemental content.

In one embodiment, each media stream's encyclopedia of articles depictedtherein is uploaded to an access provider, a stream annotation serviceprovider or other central hosting service. Then, an individual doing“entry” camera work on another media stream to populate the encyclopediacould capture an image of each article and enter very basic information.The image and basic information could be sent to the access provider,where the image and basic information would be compared with informationalready on file. The access provider would then send back “probable”matches that a user could simply adopt or reject. This would simplifythe process of inputting the initial information into the encyclopedia.

In one embodiment, the encyclopedia of articles appearing in a mediastream is uploaded to an access provider prior to filming of the mediastream to facilitate various aspects of the product placement process(e.g., sending an email to an advertiser stating that “your article willbe appearing in a media stream—contact the access provider foradditional information regarding how you can provide targetedadvertising and immediate purchase opportunities”).

Video recognition may also be used to identify segments of a mediastream in which personalized product placements can be digitally added.For example, a video recognition system might recognize a soft drink canappearing in a media stream represents an “old” can design used by theseller of the soft drink. This knowledge could allow the seller todetermine the segments of the media stream in which a new imagedepicting a more current design of the can could be inserted.Alternatively, data mining techniques could be implemented in order todetermine optimal articles to depict with respect to a given viewer(e.g., a soft drink versus a can of beer versus a glass of milk).

FIG. 5 is a flow diagram illustrating another embodiment of a method 500for producing coded media streams. Specifically, the method 500 producescoded media streams based on audio cues in the original media stream(i.e., as opposed to video cues). That is, the method 500 allows mappingof metadata related to articles that are mentioned verbally, but may notnecessarily be depicted visually, in the media stream. Like the methods200 and 400, the method 500 may be implemented by a content creator 102in order to facilitate stream annotation by a stream annotation serviceprovider 112. Alternatively, the method 500 could be implementedentirely at the stream annotation service provider 112.

The method 500 is initialized at step 502 and proceeds to step 504,where the method 500 creates or obtains an “encyclopedia” (i.e., adatabase) of one or more articles and/or individuals (e.g., actors) thatwill potentially appear in a media stream. In one embodiment, a databaseentry for an article or individual includes at least one of: anidentifying name for the article or individual, one or more images ofthe article or individual, textual identifying information about thearticle or individual, an identification number that uniquely identifiesthe article or individual, an indication of whether the article orindividual is likely to be moving in the media stream (e.g., a car) orstatic and other data characterizing certain attributes that may enhancethe accuracy of a video recognition system (e.g., luminance, hue orreflectivity). In another embodiment, a database entry for an article orindividual could be obtained by comparison to entries in anotherencyclopedia, simplifying this initial step. In another embodiment, thedatabase is further populated by automatically generating a range ofpotentially relevant supplemental database entries whenever a new entryis created (e.g., a new entry for the Eiffel Tower automaticallytriggers generation of entries associated with Paris, such as an entryfor Notre Dame).

In step 506, the method 500 obtains a segment identifier-encoded mediastream (i.e., a media stream whose segments have already been encodedwith identifying information). The method 500 then proceeds to step 508and processes the media stream in accordance with a speech recognitionsystem, using the encyclopedia created in step 504 as an input tofacilitate recognition. The relatively small size of the encyclopediashould enable more accurate recognition of articles mentioned in themedia stream.

In step 510, the method uses the recognition results to associate one ormore segment identifiers of the media stream with one or more article(or individual) IDs for articles (or individuals) appearing in thecorresponding segment(s). In one embodiment, association of segmentidentifiers and article (or individual) IDs is contingent upon humanverification of the recognition results. Thus, an index of articles (andindividuals) to segments is created. This index may then be provided,for example, to the stream annotation service provider 112 for furtheruse and processing, as described in greater detail below. The method 500is then terminates in step 512.

In further embodiments of the method 500, the recognition processingthat is performed on the media stream (i.e., as in step 508) isperformed in accordance with closed captioning information (i.e., a textsearch) rather than with audio information.

In a further embodiment still, the tagging techniques of the method 200could be combined with the method 500 for enhanced recognition results.For example, a tag ID could be used to verify audio or text recognitionprocessing results.

FIG. 6 is a flow diagram illustrating one embodiment of a method 600 forannotating media streams, according to the present invention. The method600 may be implemented, for example, by an entity that produces or editsmedia streams for release (including release by traditional televisiondistribution means or by distribution on a network, including theInternet). For instance, the method 600 may be implemented in itsentirety by the stream annotation service provider 112 of FIG. 1.Alternatively, the method 600 may be implemented by the streamannotation service provider 112 in conjunction with a content creator102.

The method 600 is initialized at step 602 and proceeds to step 604,where the method 600 obtains a coded (i.e., indexed) media stream, e.g.,as created by the content creator 102. As discussed above, at least aportion of the coded media stream is logically divided into a pluralityof identified segments comprising one or more frames of the mediastream. These segments are further associated with one or more articlesappearing therein.

In step 606, the method 600 maps at least one of the segment identifiersto at least one item of supplemental content or metadata (e.g., dataabout the media stream that is underlying the video signal). In oneembodiment, the supplemental content comprises information (such ascommercial information or documentary sources) relating to one or morearticles that appear in the media stream. As discussed above, thearticle may be a commercial product appearing in the media stream (e.g.,a sweater worn by or a beverage consumed by an actress), an individualappearing in or associated with the media stream (e.g., an actor ordirector) an audio signal that is part of the media stream (e.g., asoundtrack song) or a “concept” suggested by at least a portion of themedia stream (e.g., a beach scene may suggest a vacation, a restaurantscene may suggest restaurant reservations, or a media stream comprisinga film might suggest obtaining tickets for a screening of a sequel). Infurther embodiments, the supplemental content may comprise coupons forcommercial products. Redemption of such coupons may be enabled, forexample, by allowing a viewer to store a given segment identifier from amedia stream, which will trigger the storage of a token under theviewer's account with a service provider. In one embodiment, the tokenmay be printed and presented to a seller. To prevent using the sametoken for multiple different purchasers, data regarding a transactionconsummated in a connection with the token may be recorded.

Thus, for example, the item of supplemental content may be a filmographyof an actor appearing in the media stream (or listed in the credits), alist of albums released by a band heard in the media stream, information(such as a real-time World Wide Web hyperlink) about a commerciallyavailable product appearing in the media stream, or information about atravel service where an image of a location (e.g., a beach, mountains,etc.) is depicted, a smell representative of an article (e.g., aperfume) appearing in the media stream or even a physical responserelated to the media stream (e.g., similar to the feedback given byvibrating video game controllers). In one embodiment, any given segmentidentifier identifying a segment of the media stream may be mapped tosupplemental content relating to one or more articles appearing in themedia stream.

In one embodiment, the mapping associates items of supplemental contentwith segment identifiers that identify segments that convey informationthat is relevant to the items of supplemental content (e.g.,supplemental content relating to a particular brand of soft drink mightbe mapped to a segment identifier identifying a segment that in whichthe soft drink is depicted or in which an actor mentions the name of thesoft drink). In another embodiment, the mapping associates items ofsupplemental content with segment identifiers identifying segmentsassociated with menus or summaries of articles that are depicted ormentioned in various other segments (e.g., “Articles appearing in thefollowing media stream and for which additional information is availableinclude x, y and z”). In another embodiment, the supplemental content isassociated with the entire media stream (all segments, e.g., as in thecase of a film soundtrack or supplemental content related to purchasingalternative versions of the media stream, such as a DVD of the mediastream or tickets to a theatrical screening of the media stream).

FIG. 21, for example, illustrates an exemplary mapping 2100 in whichsegment identifiers (“XYZ-100” through “XYZ-211”) identifying segmentsof the media stream 2000 of FIG. 20 are mapped to items 2102 ₁-2102 _(n)of supplemental content (hereinafter collectively referred to as “items2102”). The items 2102 relate to articles (e.g., clothing, furniture)appearing in the segments of the media stream 2000. For example, theitem 2102 ₁ for a men's shirt indicates that the shirt is related to thesegments of the media stream 2000 that are identified by segmentidentifiers “XYZ-100” through “XYZ-102”.

In one embodiment, mapping of an item of supplemental content to asegment identifier includes mapping the item of supplemental content toa segment identifier identifying a number of frames occurring after thesegment in which the associated article appears (i.e., temporally). Thisprovides a time delay that, as described in greater detail below, allowsa viewer some time to react to the depiction of an article in the mediastream in which he or she may be interested. In one embodiment, the timedelay is adjusted in accordance with one or more parameters that may beuseful in compensating for the reaction time inherent in a specificviewer signifying an interest in accessing supplemental informationabout a depicted article, or in compensating for the occurrence of quickcuts in the media stream (i.e., where articles appearing in the mediastream may change rapidly).

In further embodiments, the supplemental content includes informationuseful in performing data mining. For example, FIG. 22 illustrates anexemplary mapping 2200 in which segments of the media stream 2000illustrated in FIG. 20 are associated with supplemental content such as:segment genre, the scene depicted by the segment, the level of actionoccurring in the segment, actors featured in the segment, music featuredin the segment and the type of “shot” represented by the segment. Forinstance, as illustrated, segment identifier XYZ-100 is mapped tosupplemental content indicating that the associated segment of the mediastream 2000 represent the “comedy” genre, includes a scene of “banter”and features “Actress A,” among other information.

In step 608, the method 600 stores the mapping of segment identifiers toitems of supplemental content. In one embodiment, the mapping is storedin a store (e.g., database 2104 of FIG. 21) that is indexed according toa unique program identifier for the media stream. This programidentifier identifies the entire media stream, in contrast to thesegment identifiers which identify individual segments of the mediastream.

The method 600 is terminated in step 612.

Mapping the segment identifiers identifying segments of the media streamto supplemental content is computationally simpler than conventionalproduct placement markup techniques, such as those that embedsupplemental content in the media stream itself, because it requiresminimal special coding to implement. Accordingly, it is expected thatthe technique embodied in the method 600 is more cost effective thanknown techniques for associating supplemental content with mediastreams. Moreover, the simplicity of the present approach affordsgreater flexibility in terms of when supplemental content may beassociated with the media stream and what kinds of supplemental contentcan be associated. For example, a business providing the association ofsupplemental content may provide such association for a limited periodof time in exchange for appropriate consideration, and thereafter removethe association without having to re-code the media stream itself.

In one embodiment, the method 600 allows a content creator and/or mapperto visually review the supplemental content mapping associated with themedia stream, so as to confirm that items of supplemental content aremapped to the proper segment identifiers. In one embodiment, this isenabled by a split screen display where a first portion of the displaydepicts the media stream and a second portion of the displayautomatically depicts all of the items of supplemental content that areassociated with the currently depicted segment of the media stream. Inone embodiment, all of the associated items of supplemental content thatare associated with a given segment of the media stream, regardless ofviewer type, are displayed simultaneously (e.g., items of supplementalcontent that will be shown to males and items of supplemental contentthat will be shown to females; the “hip hop” version, etc.). In oneembodiment, the items of supplemental content are presented in varioussummary formats, such as a written descriptor (i.e., without an image ofthe supplemental content), a “thumbnail” image, and other similar means.In one embodiment, all of the items of supplemental content that wouldappear to a viewer are shown to a user reviewing the mapping, so thatthe user can easily review the overall viewer experience (e.g., verifythe order in which the items of supplemental content appear on a viewerdisplay). In one embodiment, the user is allowed to review all items ofsupplemental content that include a particular element of information(e.g., a particular article), so that uniformity across various items ofsupplemental content can be preserved and/or differences among items ofsupplemental content can be confirmed as being intended.

In one embodiment, for training or other purposes, an individual orgroup of individuals creating a mapping (e.g., content creators,collaborators, a third-party stream annotation service provider or thelike) is presented with a display that includes the media stream and acoding produced by an expert, so that the individual(s) creating themapping can view the actual coding (i.e., including both thesupplemental content to be displayed to viewers and information that isnot intended to be displayed to viewers). In one embodiment, the codingis not presented until after the individual(s) creating the mapping hasmade his/her selections of supplemental content for the mapping. In oneembodiment, mistakes made by the individual(s) creating the mapping(e.g., deviations from a coding produced by an expert) are tracked, inorder to show the individual(s) creating the mapping examples of his/herproblematic coding, so that he/she has additional opportunities topractice coding techniques that are more difficult for him/her. In oneembodiment, the approach described above may be utilized for qualitycontrol purposes, whereby an expert reviews and confirms, declines, ormodifies mappings created by the individual(s) creating the mapping.

An access provider or stream annotation service provider may find ituseful to ensure that all mappings utilize a common approach withrespect to substantive aspects of “what” should be mapped (e.g., for howmany segments must an image appear before the image is logged), “how” itshould be mapped (e.g., “action” versus “romance”), and the actualnotation for the mapping (e.g., “romance” versus “romantic”). Tofacilitate the commonality of approach, the individual(s) creating themapping may need to be trained to a specific standard. In oneembodiment, one or more individuals (e.g., employees of an accessprovider or stream annotation service provider) view a media streamsimultaneously and create and map appropriate supplemental contentand/or add appropriate information regarding the subject matter or otherinformation about the media stream (e.g., log a particular segment of amedia stream as “action” instead of “romance”). In one embodiment, anumber of individuals create a mapping for the same media stream, but afinal mapping is determined by a majority choice of the individuals(e.g., two individuals designated one segment of media stream as“action” while eight individuals designated the same segment as“adventure”). Alternatively, mapping performed by any number ofindividuals (potentially utilizing the ability of the Internet to easilyallow multiple individuals to collaborate on the same project) arepresented in bulk to another individual who “picks and chooses” fromamong the proposed mappings to select the final, authoritative mapping.In one embodiment, the mapping is then provided in whole or in part toone or more third parties for review and comments (e.g., to the contentcreator who created the media stream, to advertisers whose supplementalcontent is included in the mapping, etc.).

In one embodiment, each media stream is divided into segments with alog-in system to ensure that no one individual collaborating on thecreation of a mapping sees more than one segment from the same mediastream with which the mapping is associated. This may be useful, forexample, for avoiding “burnout” of the individuals creating the mappingor to avoid revelation of plot points.

In one embodiment, an online system enables teams of individuals tocollaborate in creating the mapping. In one embodiment, a log-in/log-outsystem ensures that all mapping work is maintained, but with changesshown from prior mapping sessions. In one embodiment, the individualscreating the mapping are provided with the capability to leave notes foreach other regarding decisions that they make with the mapping.

FIG. 7 is a flow diagram illustrating another embodiment of a method 700for annotating media streams, according to the present invention. Inparticular, the method 700 allows content creators, sellers or ad buyersto annotate a media stream by creating mappings of segment identifiersto supplemental content (e.g., mapping by an intermediate party is notnecessary). In one embodiment, a content creator who annotates a mediastream in accordance with method 700 is an individual (e.g., as opposedto a studio or broadcaster), or even a viewer. Thus, in one embodiment,the method 700 is implemented at the stream annotation service provider112 or at an access provider 106. Although further discussion ofembodiments that allow users to create and annotate their own mediastreams will be discussed in the context of a service provided by anaccess provider, those skilled in the art will appreciate that suchapplications may alternatively be provided by, for example, a streamannotation service provider or other types of service providers.

The method 700 is initialized at step 702 and proceeds to step 704,where the method 700 receives a media stream. The media stream may beprovided, for example, by a content creator 102. In one embodiment, themedia stream is a final media stream (i.e., where raw footage has beensubjected to editing and other production steps). In one embodiment, themedia stream is received in a digital format, such as MPEG. In oneembodiment, receipt of the media stream involves transcoding from afirst file format to a second file format (e.g., from MPEG to Flash, avideo format supplied by Adobe, Inc. of San Jose, Calif.). In oneembodiment, the receipt of the media stream also includes the receipt ofmetadata associated with segment identifiers of the media stream (e.g.,indicating, for example, scene changes, “mood” changes, shot changes, orthe like). This information may be useful later in selectingsupplemental content to associate with the identified segments.

In one embodiment, the method 700 receives the media stream insubstantially real time (i.e., as the media stream is being filmed).This would allow the method 700 to inform sellers when their articlesare being filmed (e.g., by sending an automated email notification to apre-registered account), if the article is tagged in accordance withsome sort of recognizable tagging scheme (e.g., an RFID tag, asdiscussed above). This also allows the “rough cut” of the scene to bedisplayed to seller, so that the seller can see how an article is beingused and determine if it wishes to pay for more substantial advertising(i.e., in the form of mapped supplemental content).

In step 706, the method 700 receives supplemental content that isassociated with the media stream (e.g., as dictated or requested by thecontent creator). In one embodiment, such associations are made withrespect to temporal segments of the media stream (e.g., segmentidentifier X, which identifies the first x minutes of the media stream,is associated with supplemental content Y). In another embodiment, theassociations are made with respect to metadata included in (or providedwith) the media stream (e.g., segment identifiers identifying segmentsdefined by segment identifiers, cue points or the like). In oneembodiment, multiple items of supplemental content that are associatedwith a common segment of the media stream are associated as a set,rather than associated individually. Thus, the supplemental contentassociated with a given segment of the media stream may comprise as fewas one item of supplemental content, or as many as a plurality ofindividual items of supplemental content.

In one embodiment, the supplemental content comprises any form ofcontent, including, but not limited to: text, audio, video, multimediapresentations, photographs or other images, uniform resource locators(URLs), computer applications, data, commentary and advertising copy.

In further embodiments, supplemental content includes elements thatenable interactive participation by a viewer and that may be initiatedby the viewer using one or more modalities (such as pressing a key orbutton on a computer mouse, keyboard, or other input device; a verbalrequest processed by speech recognition technology; a signifying motionprocessed by a motion recognition system or the like). One example ofsupplemental content that enables such viewer interactivity issupplemental content that enables a viewer to signify an interest inreceiving secondary information or services related to the primarysupplemental content (e.g., “Click here to learn more about this DVD andto receive a discount coupon”). Other examples include enabling a viewerto register a vote on a particular issue, to activate “short cut”controls over a display device (such as controls that allow advancing ofthe media stream to a specific scene, e.g., “Click here to skip forwardto the car chase” or “Click here to ‘see that again’”), or to view areplacement or complementary media stream either in a primary or in asecondary viewing screen (e.g., replacing the media stream with analternate media stream or presenting an alternative media stream in asupplemental viewing screen).

In further embodiments still, supplemental content includesinstructions, commands, or other information that is intended to beprocessed or activated automatically upon the occurrence of a particularevent. For example, such instructions may “pause” the media stream for aset period of time (e.g., to allow the viewer to view the supplementalcontent without being distracted by the media stream) or “lock out” thefast-forward functionality of the display device until a specific itemof supplemental content has completed certain processes (e.g.,presenting an entire animation), or until the viewer has made a specificselection (e.g., answering “yes” or “no” in response to a question).Other examples include instructions for utilizing a specific “skin”(i.e., a specific graphical “look and feel”) for the display device ormedia player or for activating or directing the presentation of certaininformation or applications that are independent of the supplementalcontent (such as opening the viewer's email system).

In some embodiments, supplemental content includes keywords, tags,instructions or other identifying or indexing information that may beassociated with the media stream but that is intended to identify oractivate an independent process or system, such as may reside on a thirdparty website. For example, if a unique identifier (perhaps a taxidentification number, URL, or another number assigned by the accessprovider) signifying a certain brand of snowboard manufacturer has beenembedded within the supplemental content, when the media stream isviewed on a website, the appropriate processing system at that websitemay process the identifier in order to select and display banneradvertising related to that specific snowboard manufacturer.

In some embodiments, supplemental content includes keywords, tags,instructions or other identifying or indexing information that isintended to assist in the identification and retrieval of subject matterof interest. Many methods currently utilized for subject matter taggingof this type are less than optimal due to the fact that the tags aregenerally applied to an entire media stream rather than morespecifically applied to only the relevant segments of the media stream.For example, a user uploading a thirty minute home video of a trip toAfrica to a video access provider and utilizing generalized terms todescribe the subject matter of the entire video would likely tag suchvideo as “Africa” and “travel,” whereas much of the value of such to theviewing community may be in the fact that the video contains threeminutes of Kilimanjaro at sunset footage, six minutes of NgorongoroCrater footage, thirteen minutes of Serengeti footage (of which tenminutes are focused on lions and three minutes are focused on giraffes),etc. The present invention provides a superior approach to taggingbecause tags can be applied as specifically as frame-by-frame toidentify the information being conveyed by a media stream.

Further embodiments of the invention allow a user to identify specificcategories of content tags, so as to aid in the identification ofspecific types of information. Such categorized tagging improves searchfunctionality by enabling the ability to search for specific types ofcontent by differentiating tags that are derived from or relate to theaudio portion of a media stream (e.g., the subject matter beingdiscussed in a video) as “audio-derived” tags, the images appearing inthe media stream as “image-derived” tags, the general mood of the mediastream as “mood-derived” tags, and the like. This would enable a viewerto search, for example, for an audio-derived tag on the topic of “globalwarming” with the image-derived tag regarding Kilimanjaro. In oneparticular embodiment, a “thumbnail” representation of the specificmedia stream segment to which the intra-stream content tag applies canbe generated (e.g., a small image of a frame of the video presentingKilimanjaro), and this thumbnail may be presented to other users in thesearch results that they obtain.

Further embodiments of supplemental content include data useful forfacilitating commercial transactions, such as between or among a contentcreator, a viewer, an access provider, a contributor of certainsupplemental content, and/or the owner of a website or service on whichan annotated media stream is hosted. For example, a content creator maywish to include a unique identifier that identifies a particular mediastream or a particular item of supplemental content as having beenprovided by the content creator (e.g., perhaps utilizing an accountnumber associated with the content creator) so that the content creatormay be properly “credited” by an access provider for having made suchcontributions. Other examples include information relating to paymentmechanisms (e.g., the content creator's bank account routing number),commission information (e.g., the specific percentage fee to be paid tothe content creator if a viewer completes a transaction via thesupplemental content contributed by the content creator), feeinformation (e.g., the fee to be paid to the content creator for eachtime a certain item of supplemental content is viewed), and the like.

Other embodiments of supplemental content include variable data, such asa data table that records the number of times that a given item ofsupplemental content has been presented. For example, each time that amedia stream is displayed for the amount of time necessary to haveactivated the presentation of a particular item of supplemental content,a variable data field may be updated to register the “viewing”. Thisdata could then be made available to other systems within an accessprovider, facilitating detailed tracking analysis and reporting (such astracking the number of times that a particular item of supplementalcontent has been viewed by a particular viewer during a specific timeperiod).

In some embodiments, supplemental content includes descriptors of thenature of the annotated media stream, such as descriptors identifying“family friendly” media content, “weapons related” advertising content,“adult themed” content, and the like. This information may be used byindividual viewers or others or by automated filtering systems and thelike to restrict viewing or other access to such types of information.

In one embodiment, supplemental content is intended to be visible to aviewer of the annotated media stream (as would be the case with respectto most advertising copy). In another embodiment, supplemental contentis intended to be only indirectly apparent to the viewer (as would bethe case with some of the “display device or media player control” typesof supplemental content described above, in which only the impact of thesupplemental content upon the media stream would be evident to theviewer). In yet another embodiment, supplemental content is intended tobe invisible to the viewer (as would be the case with respect tosupplemental content consisting of subject matter identifiers,transaction facilitation data, and the like).

All of the types of supplemental content described above may be broughttogether in various combinations. For example, a composite item ofsupplemental content could be created by combing subsets of individualitems of supplemental content for: (1) pausing the media stream; and (2)and allowing a viewer of the media stream to register a vote on aparticular issue.

In one embodiment, a first item of supplemental content may be madedependent upon a second item of supplemental content. For example, if afirst item of supplemental content conveys that the media stream isintended for an adult audience, then a second item of supplementalcontent may be programmed to present a particular type of advertisement.In further embodiments, items of supplemental content may also be madedependent upon viewer interaction with a first item of supplementalcontent. For example, if a first item of supplemental content is acomposite item created by combining individual items of supplementalcontent for pausing the media stream and for allowing a viewer toregister a vote on a particular issue, then a second item ofsupplemental content could provide instructions for fast-forwarding themedia stream to the portion of the media stream that is most relevant tothe vote registered.

In further embodiments, supplemental content can be specifically createdfor access provider purposes (e.g., information placed with the accessprovider regarding a specific product or uniform resource locator (URL)hyperlink to a custom e-commerce store). Alternatively, supplementalcontent can be created to showcase a particular set of articles, such asarticles created and/or offered for sale by the content creatorhim/herself (e.g., a garage band's own DVD). In another embodiment,supplemental content includes pre-existing information or services thatare independent from the access provider (e.g., an entry in a public webencyclopedia or an e-commerce website).

In one embodiment, primary supplemental content is linked to secondarysupplemental content comprised of the same range of information and/orfunctionality as the primary supplemental content. Such secondarysupplemental content may be intended, for example, to be presented byvarious means such as by replacing the primary supplemental content(i.e., a “roll over”, such as when a viewer cursors over a particulararea of the viewing screen and additional information is presented), bypresenting the secondary supplemental content in a separate viewing areaor by opening a separate web browser to a relevant web page.

In one embodiment, the supplemental content is created by the creator ofthe media stream. In another embodiment, the supplemental content isselected by the creator of the media stream from a library ofsupplemental content (e.g., provided by the stream annotation serviceprovider 112 or by a media stream access provider). In one embodiment,the library of supplemental content includes pre-approved advertisinginformation. A library of supplemental content may consist of any numberof entries (i.e., for specific articles, people, concepts, etc.), andeach entry may in turn be comprised of any number of items of relevantsupplemental content. Each entry and/or item of supplemental content mayconsist of the same range of information and/or functionality ascomprises the supplemental content described above. For example, alibrary entry for a given article may include commercial and/ornoncommercial images of the article, textual or audio information aboutthe article, a short “commercial” about the article, a hyperlink to buythe article in real time, a retail price of the article or to otherthird-party applications or other advertising. This content may beobtained through an affiliation with sellers 100 and/or ad buyers 108.

In further embodiments, a library entry for an article includes anamount of money that a provider of the article (e.g., a seller) iswilling to pay to have the article depicted in a media stream and/or tohave the article associated with supplemental content via a mapping(e.g., possibly subject to specified factors such as media streamcontent or genre or the like). This would allow a content creator to“shop” according to certain parameters in order to identify articlesthat will bring the greatest commercial value. For example if thecontent creator knows that an actor will be depicted wearing sunglassesin a particular segment of a media stream, the content creator cansearch for all types of sunglasses listed in the library and see therelative value of filming the actor wearing specific pairs of sunglasses(e.g., brand X is willing to pay more than brand Y to have theirsunglasses placed) or alternatively filming the actor wearing a“generic” pair of sunglasses that would potentially facilitateassociation of the media stream with supplemental content relating toany number of brands of sunglasses. In one embodiment, the method 700may provide an auction-type environment in which sellers can bid forplacement of an article in a media stream. For example, FIG. 44illustrates an exemplary scenario 4400 in which the ability to providean item of supplemental content relating to a given article 4402 (e.g.,a ski hat) appearing a media stream 4404 is auctioned off to varioussellers. As illustrated, the item of supplemental content contributed bythe seller who is willing to pay the most to have their article placed(e.g., Seller B) is selected for the final mapping 4406.

In further embodiments still, a library entry for an article includestracked data regarding the popularity of the article with viewers ofannotated media streams (e.g., number of click-throughs receivedper-article or per-media stream). This would allow content creators tocreate associations between media streams and supplemental content moreeffectively. In a further embodiment, a library entry for an articlerecommends other similar or alternative articles. This may help contentcreators to, for example, identify more effective articles for depictionin a media stream (e.g., supplemental content for article X attractstwice as many viewers as supplemental content for article Y), or toidentify other effective associations that may not have been considered(e.g., viewers that view supplemental content for article A also tend toview supplemental content for article B). In yet another embodiment, alibrary entry for an article includes a demographic identifier (e.g.,“young and hip”, “golden oldie”, “teen rebel”, etc.). These demographicidentifiers may be automatically provided, or may be customized by auser of the library.

In one embodiment, the library is organized for easy search/browsing bya number of methods. For example, content in the library may beorganized according to: a hierarchical structure (e.g., men's clothing,sportswear, skiwear, jackets, down jackets); keywords (e.g., brand orarticle name); most recent additions/uploads; most popular articles(optionally organized by demographics such as age, gender, affinitygroup, etc.); a percentage commission that an associated seller iswilling to pay (e.g., x dollars to have content listed first); or assuggested by a description of the media stream (e.g., keywords). In oneembodiment, the library allows a user to select supplemental content foractual or potential association with the media stream by at least one ofthe following methods: drag and drop, check box, article number entry orarticle number selection.

In one embodiment, the library of supplemental content is “live linked”with third party sources from which the supplemental content is derived(e.g., sellers, such as seller 110 of FIG. 1), thereby allowing thelibrary to remain as up to date as possible. For example, libraryentries containing items of supplemental content contributed by thirdparty sources may be automatically updated in response to changes in thecontributor's product inventory, pricing, or marketing. Such“placeholders” may be populated, for example, by using automated “bots”to obtain information from third party web sites (e.g., pulling allguitars for sale on an online classifieds site) or by referencingremotely hosted databases through APIs provided by the data owner (e.g.,accessing an online auction site's “current auctions” database through aweb services API).

In one embodiment, a third party source of supplemental content mayspecialize in providing particular types of supplemental content, suchas “discount”-oriented content, “luxury”-oriented content, “locallyavailable”-oriented content or foreign language-oriented content.

In one embodiment, a portion of the library that is relevant to thearticles of a given seller (e.g., merchant, supplier or manufacturer) ismade accessible to the seller through a password secured system thatallows the seller to securely update its respective portion of thelibrary. In a further embodiment, articles having entries in the librarymay be linked to articles that have been filmed and are currently beingpresented to viewers by an access provider at which the method 700executes (as described in greater detail below), so that active mappingscan be updated to reflect new information. In one embodiment, when aseller removes an entry relating to a particular article from thelibrary, a notice (e.g., created automatically or by an individualauthorized by the seller) is sent to content creators who have providedmedia streams that are associated with the removed article. In oneembodiment, any annotation in the media stream that is associated withthe removed entry is automatically rendered “invisible” or inaccessibleto viewers of the media stream (but the association is not removed ormodified until approved by the party that provided the media stream).

In one embodiment, the individual items of supplemental contentcontained in a library entry for a given article may comprise variouscategories of information such as raw information (e.g., the specificprice of a given article, the specific name of the article, such as“Brand X MP3 Player—3 GB,” a specific URL for purchase fulfillment orthe like), stylized information (e.g., the name of the given articlepresented using several different fonts, or several different images ofthe same article), composite content (e.g., combinations of individualitems of supplemental content, such as raw information and stylizedinformation, combined to form a single item of supplemental content,such as a stylized “Buy Now” image that includes an embedded URL toenable a viewer to access a purchasing opportunity), or compiledinformation (e.g., various combinations of raw, stylized, and compositeinformation that form a comprehensive entry).

An entry within a library may include functionality that is not directlyrelated to the selection or creation of specific items of supplementalcontent. For example, within a given library entry, a library user maybe enabled to view a summary (perhaps via “thumbnail” representations ofmedia streams or some other form of summary data) of all media streamswith which certain items of supplemental content have been deployed(i.e., a visitor to a specific library entry will be able to view allmedia streams deploying some or all of the items of supplemental contentcontained in the library entry).

In one embodiment, library entries include “placeholders” thatfacilitate the assignment of any of a number of differentcharacteristics or data to an individual item of supplemental content.For example, a number of different sellers may offer the same brand ofshoe, and in order to simplify the library for users by “grouping” likeentries, a service hosting the library may choose to list the particularshoe under a single library entry. However, because of the number ofsellers that can provide that shoe, the entry needs to be able toaccommodate the multiplicity of sellers that have identified themselvesto the library host as providing that shoe. To facilitate thiscapability, the library entry may include a “placeholder” that iscomprised of or is linked in some manner to any number of equivalentitems of supplemental content. In turn, the equivalent items ofsupplemental content may be placed within a prioritization hierarchy sothat certain of the equivalent items of supplemental content havepriority, perhaps as determined by reference to specific rules orinstructions. For example, in the shoe example described above, aplaceholder relating to “Seller's Name” may reference three equivalentitems of supplemental content: the names of Seller 1, Seller 2, andSeller 3. The three seller names could be placed in a prioritizationhierarchy such that Seller 1 is displayed as the “Seller's Name” as adefault when a user accesses the library entry (and/or when that libraryentry is embodied within a specific item of supplemental content),unless certain other conditions are met (in which case one of the othersellers' names is displayed). A user accessing the library entry mayaccess, select, and utilize a single item of supplemental content(perhaps even without an awareness that multiple different equivalentitems of supplemental content are linked to an associated placeholder)in the creation of a mapping. The equivalent items of supplementalcontent may also consist of multiple variations of the same basic itemof supplemental content, such as multiple photographs of the sameproduct shot from different camera angles. In such cases, the user maybe enabled to select a particular one or more of the equivalent items ofsupplemental content.

In another embodiment, more than one library entry may be made availablein the library with respect to a particular product, such as relating toa single product that is offered by multiple different sellers atdiffering prices. The library entries may be organized and groupedaccording to various criteria (e.g., supplemental content including “lowprice” or “in stock now” advertisements, supplemental branding contentas opposed to advertising content, supplemental content with or withoutmusic, graphics heavy supplemental content, supplemental contenttargeted at particular demographics, geography or languages, and thelike). As discussed earlier herein, library entries and items ofsupplemental content contained therein need not comprise advertising orcommercial material (e.g., they may consist of items of artisticexpression, software “mini-apps,” etc.), but may, nonetheless, beincluded in the hierarchical organization of the library, for exampleunder more abstract or general headings such as “subtitles,”“animations,” or “photos.”

The items of supplemental content contained in a given library entry maycome from any number of sources including from the service hosting thelibrary, from users of the library, from specific advertisers, or from“affiliate” programs of online merchandisers and aggregators. Inclusionof library entries or items of supplemental content included therein ina library and/or the “placement” prioritization of library entries maybe made dependant upon reference to various prioritization hierarchies,such as a hierarchy based on a fee that is negotiated between the accessprovider and the contributor of a given item or group of items ofsupplemental content (e.g., the manufacturer of a featured product whowishes for the library entry for the product to be prominently featuredto a certain class of library users). Such fees may be charged directly(e.g., a flat fee for the library entry to appear at the top of allsearch results within a specific category) or indirectly (e.g., if aseller has agreed to pay a relatively higher flat-rate and/or apercentage commission to the access provider with respect to eachproduct sold to a customer who was directed to the seller's site by theaccess provider, then the access provider's prioritization hierarchy forpresenting library entries may take that information into account).

A library, or portions thereof, may be created or assembled by anynumber of organizations or individuals, potentially includingadvertisers, “artists,” user groups formed within the access provider,and the like. For example, a particular manufacturer may create alibrary populated solely by library entries relating to themanufacturer's product offerings. A seller may create a librarycomprised solely of library entries relating to articles available inthe seller's specific stores. “Celebrities” or others may createlibraries comprised of library entries of interest to them (e.g., a rockstar may create a “My Favorites” library). Some libraries may begenerated by charitable organizations, and a portion of the revenuederived by the access provider or others from use of the library wouldbe shared with the charitable organization. Individual users could alsomake their own libraries comprised of library entries from other,pre-existing libraries or from their own contributions.

Any of these various types of libraries can be opened to uses by allusers, opened only with respect to certain uses (e.g., open to viewing,but not open to modification), or kept private within a group ofauthorized users. In recognition of the analysis and effort that hasbeen expended to create the library or subsets thereof, access to and/oruse of particular library entries may be offered as a premium servicesuch that users may access library entries from a given library onlyafter payment of a fee or after satisfying certain conditions (such ascreating or distributing a media stream that has generated a certainlevel of viewer popularity). In addition, the creator of a library maycharge a fee to others who wish to copy library entries for inclusion intheir own libraries (e.g., Rock Star A may charge a 1% commission on alltransactions that are derived from library entries that are included inher branded library), or the creator of a library may impose certainother restrictions, such as eliminating all library entries (orindividual items of supplemental content contained therein) that fail tomeet certain success parameters (e.g., library entries that are accessedor used by fewer than a specified percentage or number of users).

An access provider may provide tools to assist users in creatinglibraries (e.g., sub-libraries of the access provider's morecomprehensive library), for example by providing these tools in the formof downloadable software programs to be used on a user's system or as ahosted service (provided by the access provider or by a third party)that is accessed over a network. For example, the access provider mayprovide tools that enable a user to create a “home page” within a largerlibrary by enabling the user to customize “his” library by selectingparticular background colors, pictures, and other information desired bythe user.

The access provider may provide tools to assist users in creatinglibrary entries, for example by providing these tools in the form of oneor more software programs to be used on a user's system or as a hostedservice (provided by the access provider or by a third party) that isaccessed over a network. Such tools may include various means to ensureconsistency among data that is contributed on a self-service basis byvarious users. For example, such tools may provide menus or otheruniform presentations of “template” library entries that include fieldsfor supplemental content items of various specific categories to becompleted by the user (e.g., fields for seller name and price, for whichthe user must provide the specific name and dollar amount). Further,such templates may be utilized with respect to specific items ofsupplemental content, such as uniform items for “product_picture,”“retail_price,” and the like. By preserving consistency of this sort,the access provider is enabled to provide methods and processes that maybe used by and shared among all users of the service, but that stillpermit individual creativity. In addition, such tools may allow users tocreate “live links” between multiple items of supplemental contentand/or data provided by sources independent of the access provider. Bymaking available a common set of Application Programming Interfaces(APIs), the access provider can enable users to embed certain items ofsupplemental content in a library entry (e.g., a retail price itemwithin a product advertisement item), and such items of supplementalcontent would be able to access the source of such data within theuser's or a third party's systems that have been enabled to interact viasuch APIs and/or other input parameters. Also, the tools may providemenus or other uniform presentations of “classification” determiners toenable the library entries to be searched, such as identifying thecategory of a specific entry (e.g., athletic shoe) within the ontologyof the library as a whole.

As will be evident to one of skill in the art, all aspects of thelibrary functionality described herein (i.e., hosting, creating,providing access, updating, etc.) could be provided by a third partythat makes such information available to one or more media stream accessproviders, to stream annotation service providers, to individual users,to interested third parties (e.g., advertising companies interested instudying how particular library entries are being used), and the like.

A library entry and/or the items of supplemental content comprising alibrary entry may be “locked,” perhaps by the contributor(s) of thesupplemental content, by other users, or by the access provider, so thatthe library entry or the items of supplemental content making up thelibrary entry may not be directly viewed, edited, reformulated, and/orreformatted by users or by a specific class of users (e.g., “new” usersmay be excluded from editing library entries). Alternatively, libraryentries or items of supplemental content may be held “open” for users orfor a sub-set of users to utilize as they wish (e.g., using an image ofa product, but not the price) without any restriction.

Each library entry (or individual items of supplemental content therein)may be placed within an ontology, so that information can be more easilylocated and reviewed. For example, when one of the items of supplementalcontent in a particular library entry is the “retail price” of aparticular product, this item of supplemental content can be tagged orotherwise identified by the library as being retail pricing information.The tagging of items of supplemental content may be undertaken in anynumber of ways, such as by the individual contributor via reference to aspecific set of online or offline tagging information (e.g., an“ontology list” that identifies retail price information as being codedagainst the tag “Ret_Price_Current”), by requiring all librarycontributions to be input using specific forms or templates (so thatindividual items of supplemental content are automatically tagged by thesystem, e.g., data input into the “retail price” field of the form willbe tagged as the retail price), or by using an automated technology thatutilizes analysis of the raw data to determine the characteristics ofvarious items of supplemental content (e.g., the system may beinstructed to label all numbers that follow a currency symbol “$” or “£”to be a “Retail Price”).

Library entries may consist of items of supplemental content that aredesigned to be associated with all of the other items of supplementalcontent in the library entry, so as to describe certain characteristicsor properties of the other items of supplemental content. Thisfunctionality may be enabled by embedding certain descriptivecharacteristics within one or more items of supplemental content in alibrary entry, by adding one item of supplemental content to a libraryentry that is then paired or embedded in every other item ofsupplemental content within the library entry, or even by adding an itemof supplemental content that is separate for any particular libraryentry but that is embedded or paired with all items of supplementalcontent of a certain type within the entire library. This complementarytype of supplemental content item may or may not be intended to beviewed and/or edited by users. For example, with respect to a libraryentry relating to a commercial product offering, the addition of acomplementary item of supplemental content may result in every compositeitem of supplemental content in which any individual item ofsupplemental content in the library entry is incorporated to beidentified as “commercial” supplemental content (as opposed to, forexample, artistic or commentary-oriented supplemental content). Othersuch items of supplemental content may identify “family friendly”content, “weapons related” content, “adult” content, or the like. Forexample, the access provider may reference such identifying data todetermine if the supplemental content should be “locked” (e.g., toensure that all of the items of supplemental content within an “adultcontent” entry are embedded with signifiers that identify the items ofsupplemental content as relating to adult content, so that suchsupplemental content cannot be accessed or viewed by users younger thana certain age).

If a library entry were removed for any reason from the library (or onlyfor a specific category of reasons) or, alternatively, if the libraryentry were placed on “hold” by the access provider or by the libraryentry contributor, after items of supplemental content contained in thelibrary entry were associated with a media stream that will potentiallybe presented to viewers, a notification could be sent to all users whohave provided or distributed media streams that are associated with therelevant items of supplemental content. Such notice could be performedby a person or could be automatically created (e.g., through anautomated email notice or web announcement). In such linked embodiments,“dead” library entries (i.e., library entries containing informationthat is no longer up-to-date or that is no longer provided by aparticular information source) could be automatically replaced withdefault information (e.g., information regarding the general class ofarticles or a message from the access provider stating that the libraryentry is no longer available) or could be excluded entirely from thelibrary. In each of these cases, users or various sub-categories ofusers could be automatically notified of such changes, or other changessuch as updates to a particular library entry that the user haspreviously indicated to be of interest (e.g., “Click “Yes” if you wouldlike to be notified when changes are made to this entry”). In oneembodiment, once a library entry or item of supplemental contentcontained therein is removed or otherwise placed into question, any useof the library entry or the item of supplemental content (including, inone embodiment, a composite item of supplemental content thatincorporates the given item of supplemental content) is automaticallyrendered “invisible” to viewers of the library (e.g., if item 2 isremoved, item 3 would follow item 1) or is replaced with a “default”screen, but the portion of a mapping associating the item ofsupplemental content with a media stream would not actually be removeduntil approved or modified by the creator of the media stream. Thedisplay viewed by the content creator may continue to show theassociated item of supplemental content, but the display may be markedto graphically show that the item of supplemental content is no longervisible to viewers (e.g., overlaying with a red X). Alternatively, thecontent creator may be given the option to include “alternative” itemsof supplemental content to be presented under certain conditions (e.g.,instructions to present item A of supplemental content until date X,then present item B).

Items of supplemental content that are contributed to the library maybe, either automatically or upon the contributor's request,reformulated, reformatted, or otherwise modified by various processesand for various purposes, such as ensuring that all images are of thesame data format (e.g., .gif or .jpg), ensuring that certain parametersare satisfied (e.g., the size of an image, or the maximum size font), orembedding in all photographs a watermark or other elements for thepurposes of copyright protection.

In one embodiment, the contributor of a library entry may be allowed to“open” or “lock” certain items of supplemental content within a thelibrary entry, category, or library (e.g., other contributors may beallowed only to change the stylistic items of supplemental contentwithin the library entry, but not the product-related items ofsupplemental content). For example, the initial contributor of a libraryentry may “open” the “style” items in a library entry by allowing othercontributors to provide alternative “style” items of supplementalcontent (e.g., allow the background color and fonts of certain items ofsupplemental content to be changed), but “lock” the product image,retail price, and actual ad copy by preventing other contributors fromproviding alternative items in these categories. This “lock,” “unlock,”or other types of controlled access functionality could be enabled bysimply checking a box with respect to each item of supplemental contentor template field associated therewith.

In one embodiment, the access provider or library creator(s) is enabledto establish “categories” of contributors, and the access functionalitydescribed above may be made specific to these contributor categories.For example, a “default” category of contributors may be permitted toprovide commentary about a given item of supplemental content (e.g., “Agreen background would be more effective”) or to contributealternatives, but may not be allowed to actually change any items ofsupplemental content within the library entry. Another category ofcontributors, such as contributors who have been provided with specificpermission by the access provider or who, through various actions suchas the contribution of a set number of library entries, may be entitledto “administrator” access that allows them to directly edit and/ordelete items of supplemental content within a library entry or even toedit and/or delete entire library entries. For example, a particularadvertiser who establishes an “official” library may allow onlycommentary contributions and reserve all other access to theadvertiser's employees or consultants (e.g., only employees are allowedto contribute or modify library entries). Such access may be grantedthrough an initial log-in function, or access may be granted throughvarious other means such as passwords or the like. This functionalitymay be used to “ban” abusive users, potentially with respect to aparticular library entry, a given category, or an entire library.

In one embodiment, a record of all previous instantiations of a givenlibrary entry (i.e., a “document history”) may be stored and madeaccessible. These prior versions may be allocated to a revision historythat tracks, for example, the changes made, when the changes were made,who made the changes, the status of such changes (e.g., has a changebeen approved by the relevant administrator as a permanent change), etc.In one embodiment, a revision history allows a library entry to bereturned to a prior state. This may be useful, for example, if aparticular library entry is vandalized. This feature may also be usefulto library contributors for other reasons, including facilitation of theability to schedule library entry “refreshment” according to a certainschedule (e.g., “Show all library entries that have not been modified inthe last ninety days”) or of collaborative contributions among multipleparties (e.g., “Show all changes made by User X in the last ten days”).

The access provider may allow individual users to set a “watch list”that will provide notification (e.g., via email) when certain types orcategories of changes are made). For example, the initial contributor ofa given library entry may wish to be notified of every change within thelibrary entry, whereas the contributor of a given item of supplementalcontent (e.g., the contributor of the retail price) may wish to benotified only when that specific item of supplemental content ismodified.

The portion of a library entry that is relevant to a particularcontributor (e.g., a particular merchant, supplier, or manufacturer whohas contributed product information or the photographer of a particularphoto) could be made accessible exclusively to the contributor through apassword secured system so that the contributor can maintain and controlthe supplemental content contained in the relevant portion of thelibrary entry.

Many aspects of the library and the library entries may be allocated bythe access provider to third parties based on certain prioritizationcharacteristics. Those characteristics include, among others, at leastone of: the right to contribute to the library, the categorization oflibrary entries (e.g., under how many categories will a particular itemof supplemental content be listed), the prioritization of display oflibrary entries to users of the library (e.g., which library entry foran MP3 player will appear at the top of the library search results listfor the search term “MP3 player”), or which library entries will belisted with special emphasis (e.g., with a colored border around thelibrary entry to make it more distinguishable from other libraryentries). These determinations may be made by various means, includingby charging fees (either on a per contribution or a per contributorbasis, e.g., a “partner” fee), by charging tiered commissions (e.g.,providing higher entry prioritization in exchange for higher commissionsbeing paid to the access provider), or by rewarding the magnitude and/orsuccess of prior library entries (e.g., a particular contributor'slibrary entries are awarded “preferred” status based on the aggregatesuccess of the contributor's prior contributions in fostering largedollar transactions).

For purposes such as enabling the effective location of particularlibrary entries of interest, a library entry may be characterized by anynumber of ways that may or may not relate directly to the substantivecontent of the items of supplemental content contained in the libraryentry. One example might be a characterization that does not relate tosubstantive content but that does characterize the library entries bytheir “featured product” or “retail price” supplemental content items,so that, for example, multiple library entries for a particular productcould be listed from the lowest to highest retail price. One example ofa characteristic that is only indirectly related to the substantivecontent of an item of supplemental content is the number of users whohave elected to utilize all or a portion of a relevant library entry inthe creation of specific mappings or the “effectiveness” of the libraryentry in providing supplemental content that has proven to be effectiveadvertising (e.g., identifying those library entries containing items ofsupplemental content that induce higher rates of viewer interest).

At least a portion of the library may be made available by the accessprovider to various third parties to present to their specific usercommunities. For example, a seller who has contributed a number oflibrary entries might wish to present those library entries on theseller's own website, so that visitors to the seller's website areinformed of opportunities to utilize these library entries when creatingmappings using the access provider. The access provider may make theselibrary entries available via a “live link” system in which the thirdparty is provided with an API to the library entries, or the libraryentries may be provided as “downloads” that require updating. The accessprovider may also make various combinations of entries available tothird parties who seek to create branded libraries of interest to theirusers (e.g., a site that compiles “teen oriented” or “best performing”ads).

The library may be enabled to be accessed or “searched” in any number ofways. For example, library entries of interest may be located by typingin keywords, by searching by any of the characteristics of any of thecategories of items of supplemental content (e.g., listing all libraryentries that include a photograph, that include a retail price greaterthan $10.00 and that generated viewer response in the top twenty percentof comparable library entries), by the characteristics of the libraryentry contributor (e.g., all library entries contributed by on-lineshopping sites, but exclude all library entries by Site X), or by thenewest contributions. Any of the search characteristics may be utilizedin any combination (e.g., show me library entries contributed by Store Awithin the past ten days).

Certain third parties may wish to create libraries that are not formallyconnected with the access provider library, but that can be searchedand/or accessed through the access provider. To facilitate these typesof arrangements, the access provider may provide certain informationthat these third parties may include in their websites allowing theirlibraries to be searched by the access provider. For example, a sitesuch as an online classifieds site may wish for all of its postings tobe available for annotating media streams, but the site may not wish toforce the people using the site to copy their listing information overto the access provider. Rather, the site could provide to its usersand/or embed in its posting templates the codes necessary to enablesearch by the access provider.

In one embodiment, the supplemental content mapped to a given segmentidentifier identifying a segment of a media stream (where thesupplemental content comprises anywhere between 1 and n individual itemsof supplemental content) is selected at least in part by an automatedprocess, potentially in combination with actions taken by the accessprovider or by a content creator. For example, voice recognitiontechnology may be used to generate a transcript of the verbal elementsof the media stream, and the content creator may divide the transcriptinto logical segments, so that a given logical segment is allocated tosupplemental content presenting appropriate phrases or sentences (e.g.,subtitles). Alternatively, image recognition technology may be used togenerate still images of all of commercial articles appearing in a mediastream, and the content creator may utilize these still images toassociate the media stream with product placement supplemental content(i.e., the automated process generates an image of a Brand X purseappearing in a video, and the content creator maps to a segmentidentifier identifying a segment of the video additional ad copy that iscreated by the content creator or is excerpted from a supplementalcontent library).

In one embodiment, automated tools are employed by the access provideror by others (e.g., a group of users) to populate pre-made, “template”composite items of supplemental content items by assembling one or moreindividual items of supplemental content from any number of sources.These templates may include certain combinations of “placeholders” forone or more aspects or characteristics of the composite item ofsupplemental content, such as a particular layout of informationcategories (e.g., an image to left, a URL hyperlink in themiddle-center, and another image to the right), a particular layout ofindividual supplemental content types (e.g., “Product_Name” followed by“Retail_Price”), a particular graphical format (i.e., various shades ofblue and red images) or the like.

In some embodiments, a content creator may specify a category of items(e.g., “men's, leather, belt, blue”) and allow the method 700 to proposean optimal library entry or collection of library entries to select forthe creation of a mapping (e.g., as determined by reference by themethod 700, or by the content creator, where the optimal selection ismade by reference to specific criteria dictated by the access provideror by the content creator). Criteria for selection of an optimal libraryentry (or entries) may include factors such as one or more of: thelibrary entry containing supplemental content with the highest rate ofviewer click-throughs per view when associated with similar or othermedia stream content, the library entry containing supplemental contentwith the highest number of total click-throughs, which library entry'scontributor is willing to pay the highest referral commission in thespecific context of this media stream and/or a particular viewerdemographic, or other factors of interest.

Items of supplemental content may be characterized by any numberattributes, such as the general “layout” of an item of supplementalcontent (e.g., product image on the left, ad copy on the right), thegeneral “style” of the item of supplemental content as suggested throughdifferent fonts, graphics, and the like (e.g., hip-hop, versuscountry-western style), the particular product or product category thatis featured in the item of supplemental content, the “color scheme”(e.g., “hot” colors such as red and yellow for the text and background,versus “cool” colors such as blue and green) and the like. In oneembodiment, the access provider provides tools that allow users of asupplemental content library to streamline the mapping process byallowing the users to view multiple variations of the same basiccomposite item of supplemental content (e.g., by viewing a set ofcomposite supplemental content items in which each composite item ofsupplemental content differs, for example due to changes to one or moreof the supplemental content attributes). For example, a user who iscreating a mapping using items of supplemental content found in alibrary entry relating to a Brand X MP3 player may wish to viewcomposite items of supplemental content that include the same productimage, but with multiple different resolutions, sizes, or backgroundcolors, or to view the composite item of supplemental content with textinformation regarding the product presented in multiple different fonts.Each variation may potentially be viewed simultaneously with all of theother potential variations (e.g., via a scrolling view screen), or maybe viewed individually (e.g., sequentially as in “press “Next” to viewthe next variation).

In one embodiment, the present invention provides the content creatorwith access to a directed selection process that aids in the selectionof the desired supplemental content variation (e.g., a library entry orspecific item of supplemental content therein) by prompting the contentcreator with questions and then providing alternatives that match thecontent creator's responses to the questions. For example, the contentcreator may be prompted to specify whether he or she prefers “hotter”versus “cooler” background colors for the background and “photographic”versus “graphical” images of a product, and this information is thenused to provide relevant variations for the content creator'sconsideration, further modification, and/or selection. Alternatively,the content creator may be provided with functionality to speed theselection via a “select A or B” option, so that the number of choices isquickly and efficiently pared down to the most desired choice. Forexample, initial selections may focus on the background colors, and oncethe optimal color is determined, the A/B selection process proceeds tomanaging selection of the most favored image. The tools may allow thecontent creator to select the order in which selections are made (e.g.,work through the range of image options prior to presenting thebackground color choices). Such selection tools may utilize machinelearning techniques that create options for the content creator based onanalysis of prior interactions with the content creator or with otherusers under comparable circumstances. For example, machine learningmight identify that supplemental content with given characteristics(e.g., advertising featuring peanut butter) is often deployed inconjunction with supplemental content having certain othercharacteristics (e.g., advertising featuring jelly).

In one embodiment, a content creator may map supplemental content simplyby selecting a specific library entry or a specific item of supplementalcontent of interest using any combination of: a “drag-and-drop” tool, bychecking a box, by typing in item numbers, and the like.

The access provider may provide tools to enable easy and effectivedelivery or uploading of pre-made supplemental content (individual orcomposite) from a user to the access provider. Such tools may allowvarious forms of information to be provided by the user to the accessprovider, and the access provider will utilize various processes totransform the information so that it may be utilized for additionalcustomization (e.g., performed by the user or to be consistent with theformat and attributes of supplemental content that can be presented bythe access provider to viewers). For example, the .jpg format imagecomponent of an uploaded composite item of supplemental content may beconverted to .gif format, and the particular font used for textinformation in the same composite item of supplemental content may betransformed into .gif format as well, and then both .gif files aretransformed into single composite item of supplemental content that canbe processed the media stream player used by the access provider. Thetransformation process may be aided by a menu system that allows a userto quickly identify the different items of supplemental content that arebeing uploaded, or by automated detection processes that analyze theitems of supplemental content being uploaded and determine the items'attributes. Once identification is complete, the items of supplementalcontent may undergo various processes as necessary to prepare the itemsof supplemental content for final form (that may be utilized during thesynchronization or mapping process discussed in greater detail below,and, ultimately, presented to viewers).

In one embodiment, the access provider allows users (e.g., contentcreators and other users) to create their own items of supplementalcontent. Such capability may be enabled in any number of ways, includingallowing the user to contribute items of supplemental content of his/herown making or to contribute items of supplemental content obtained froma third party. In further embodiments, the user is enabled to select asingle, unified library entry as a whole (e.g., a library entry orcomponent thereof that comprises a sufficiently complete item ofsupplemental content), or to customize an item of supplemental contentthat is provided by the access provider by adding other items ofsupplemental content from the library (e.g., to create a new compositeitem of supplemental content).

In one embodiment, users are enabled to combine items of supplementalcontent obtained from the supplemental content library in various ways,potentially along with the user's own contributions, to createcustomized items of supplemental content for mapping purposes. The usermay elect to use a composite item of supplemental content contained in alibrary entry as a whole (i.e., select an entire “pre-made” compositeitem of supplemental content) or may select various individual items ofsupplemental content from the library entry or from the composite itemof supplemental content, potentially with additional material created bythe user, to assemble a customized composite item of supplementalcontent. For example, a given library entry may be for a Brand X MP3player. The “official” library entry supplied by Brand X may containseveral images of the specific model of MP3 player, informationregarding the MP3 player's physical and performance characteristics, theprice of the MP3 player, a link to a point-of-sale website, and othersimilar information that may be relevant to a potential purchaser. Theuser may select one or more individual items of supplemental contentfrom the library entry and combine the individual items with additionalcontent of his/her creation (i.e., a graphical image, a clever saying, areference to the media stream with which the supplemental content willbe associated, etc.) to create a final composite item of supplementalcontent for a mapping. The individual items of supplemental content maybe utilized in combination. For example, a user may upload a relativelygeneric item of supplemental content that he/she obtained from a thirdparty (e.g., an item of supplemental content relating to a CD offered bya garage band), and then utilize tools provided by the access providerto supplement the generic item of supplemental content with his/her owncontent (e.g., a graphical image of the band's logo) and with items ofsupplemental content selected from several library entries (e.g., anitem of supplemental content to characterize the finished composite itemof supplemental content as “commercial”, and an item of supplementalcontent consisting of a “button” image and a URL hyperlink that enablesviewers to access a purchasing opportunity for the CD).

In one embodiment, the relevancy of automatically selected items ofsupplemental content is improved by asking the user to provide a minimalamount of information, such as the general subject matter of the mediastream as it relates to appropriate advertising opportunity (e.g., aglobal warming video might be better identified as “environmental”material rather than as “how to buy a car” material), the identity ofone or more articles appearing in the media stream (e.g., a particularhybrid automobile), and/or the demographic characteristics of theaudience that is likely to view the media stream (e.g., adults, ages 25and up). To obtain this minimal amount of information, the user may beasked for any of various forms of input. For example, the user may beasked to simply check one or more boxes from a list of choices providedby the access provider (where the category choices may be linked withspecific library entry categories). These additional inputs wouldsupport automated advertising allocation that is superior to existingmethods, because even the minimal amount of human input as describedabove focuses the automated system far more accurately than afully-automated system. Thus, by requiring minimal additional effort onthe part of the user, the relevancy of the supplemental contentultimately selected for a mapping (and, thus, the quality of the userexperience and the revenue generated) would be enhanced.

This selection process may, for example, be organized in a “menu driven”or “wizard” system such that a user is queried about the media streamcontent (e.g., genre), target audience demographics, demographics of theuser him/herself, or keywords/content tags defined by the user. An“expert” user may be enabled to access additional features that are morecomplex, but also more flexible or efficient. In one embodiment, theuser may be provided with the option to stop at any point in the inputprocess, and only the input that has been provided up to the stoppingpoint is utilized for supplemental content targeting purposes (by anautomated or other type of selection system). For example, the user maybe presented with an initial menu that includes the general category of“Seasonal Supplemental Content.” A user could simply check the boxassociated with that category and indicate that he or she does not wishto provide any additional information. Alternatively, the user couldsubsequently be presented with a sub-menu including “Christmas-relatedSupplemental Content,” then another sub-menu including “ComedySupplemental Content” (as opposed to, for example, “SentimentalSupplemental Content”), then another sub-menu including “SupplementalContent for Kids,” then another sub-menu including “Supplemental ContentWith Reindeer,” then another sub-menu including specific reindeerimages, then another sub-menu including specific background colors, andso on. At any point in this process, the user could elect not to provideadditional input, and the system would select supplemental content basedon the available information.

Alternatively, the user could input to or select from a menu ofclassification choices, such as the demographic classifications thatwould likely view the media stream (e.g., “kids” versus “adults”) or thegeneral subject matter category of the media stream (e.g., “comedy”versus “sports clip—soccer”) and the method 700 could then provide theuser with items of supplemental content featuring the best-sellingcommercial articles for that audience.

The method may also suggest additional items of supplemental contentbased on an initial item or items of supplemental content selected bythe user. In one embodiment, this suggestion is based on a comparisonwith items of supplemental content selected over an aggregation of users(e.g., “Users who selected this item of supplemental content alsoselected these other items of supplemental content”).

Each of the different identification or classification options describedabove could be utilized alone or in any combination with other options,and the output of such selection processes described above may result inidentifying either a final item (individual or composite) ofsupplemental content that is available for mapping or merely a “startingpoint” item of supplemental content that is intended to support furthercustomization. For example, the process may identify an initial item ofsupplemental content related to a particular product, but the user maybe prompted to further customize the item of supplemental content byselecting a specific image of the product from a range of possibleimages, to select a specific background color or font, etc.

One type of information that a user may wish to embed within a givenitem of supplemental content (or, potentially, such information from aparticular item of supplemental content is embedded within a data filefor the mapping-enabled media stream as a whole) is an instruction toalter certain aspects of the supplemental content based on data that isaccessed by the mapping, such as data regarding certain conditionsrelating to the supplemental content. In one embodiment, the user mayembed certain instructions within the supplemental content that will betriggered by specific input provided by the access provider for thisspecific purpose. For example, a third party website may be able todisplay a set of data that is invisible to the web site's visitors butthat “triggers” appropriate responses within supplemental content thatis viewed on the web site (e.g., such data may include the “dominantcolors” of the web site, so that attributes of the supplemental contentmatch the style of the web site; the “general age group” that accessesthe web site, such as a web site primarily targeted to teenagers; or thespecific volume at which sound effects emitting from supplementalcontent, and not from the primary media stream, will be made audible).

In another embodiment, the embedded instructions are triggered byinformation that is obtained by the access provider, without directinteraction or exchange of data with the third party web site or with auser. This information may be embedded within a template that wasutilized by the content creator to create the mapping, and it need notbe information that the user directly or even knowingly included in themapping. For example, the instructions embedded within a given item ofmapped supplemental content may require the automatic access ofinformation provided to the item of supplemental content (e.g., perhapsgenerated by the media stream player, or perhaps generated during thetranscoding stage of the method 700), such that the background color ofthe supplemental content is altered to be more consistent with eitherthe general color scheme of the temporal segment of the media streamidentified by the segment identifier to which the supplemental contenthas been mapped (e.g., if the media stream presents a night scene, avideo processing system deployed by the access provider may detect thatthe primary colors are muted, and this information would trigger thebackground color of the supplemental content to be set to “dark blue”rather than the default color of “pale yellow”). As another example, themedia stream player deployed by the access provider may “read” the HTMLdata presented by the web site to which the media stream player andannotated media stream have been deployed in order to determine thegeneral color scheme, subject matter, or audience of the web site and toadjust the supplemental content accordingly.

One item of information that a content creator may include within amapping or within a given item of supplemental content is informationregarding the specific type of viewer to whom the given item ofsupplemental content should be presented. This feature could be used,for example, in situations where, in association with a given temporalsegment of a media stream, the user prefers to show a first item ofsupplemental content to one type of viewer, and a second item ofsupplemental content to a different type of viewer (e.g., “Show Item Aif the viewer is male, and show Item B the viewer is female”). Tofacilitate this capability, the content creator may include “determinerinformation” (e.g., determinations to be made based on demographicinformation about a viewer) within an item of supplemental content orwithin a mapping including the item of supplemental content thatidentifies the viewer demographics that are intended to activate theitem of supplemental content (as opposed to other items of supplementalcontent potentially presented in association with the same temporalsegment of the media stream). One or more determiners may be included bythe content creator in a given item of supplemental content or a givenmapping by having the content creator select some combination from alist of potential characteristics offered by the access provider (e.g.,the user is enabled to insert an age range, check a box with respect togender type, establish a minimum viewer income level, etc.).

This “generic” list of determiners may be linked by the access provider(in a manner that may or may not be visible to a user) to the specificinformation that is known (i.e., generated, gathered, or surmised) aboutthe viewers within a specific context, such as a specific website. Thislinkage between a generic determiner and the counterpart to suchdeterminer in the context of a specific web site is useful, becausealthough many web sites collect data regarding specific viewers, thereis typically no uniform manner for classifying such classifications.Thus, by enabling the access provider to keep a single, updatedinventory of such classifications of data and links to relevant genericdeterminers, the correct processes are enabled at each web site. Thisalso eliminates or reduces the need for users to provide multipledifferent codes to activate the same basic processes at different sites.For example, Web Site A may have access to the income level, age andgender of its visitors (who include viewers of the media streamspresented on Web Site A), whereas Web Site B may only have access to thevisitors' age information. As another example, Web Site A may collectvisitor age data as a range of ages (e.g., twenty five to thirty years),whereas Web Site B may collect the specific age (e.g., twenty-eightyears). In such a context, the access provider would link thedemographic determiner (provided by the user) such that the specificitem of supplemental content would elicit as much information as isavailable about the viewer at each web site. For example, the determinerwould be able to process the broad age range information provided by WebSite A as well as the specific age information provided by Web Site B,while disregarding any of the other determiners with respect to Web SiteB, due to the fact that these other determiners are not enabled by WebSite A. Enabling this type of functionality is an advantage for contentcreators, because the content creators are not required to seek out theinformation for each specific web site on which their annotated mediastream may be shown (so long as the access provider has already linkedthe correct demographic determiner). Enabling this type of functionalityis also an advantage for viewers, because the viewers are provided withmore relevant supplemental content. Finally, enabling this type offunctionality is an advantage for the access provider and for the websites on which the annotated media stream appears, because it creates asuperior viewing experience and higher levels of viewer interaction,which will hopefully lead to increased monetization of the content.

In addition to the approach whereby specific attributes of a given itemof supplemental content or a given portion of a mapping may be madeconditional with respect to viewing location, device types, viewerdemographics, etc., as described above, attributes may also be madeconditional on factors such as past viewer behavior (e.g., “Has theviewer previously been presented with Supplemental Content Item A morethan X times?”), aggregate viewing statistics, or a random selection.

In one embodiment, as discussed above, a single item of supplementalcontent may include different attributes or components (e.g., multipleindividual items of supplemental content combined within a compositeitem of supplemental content), such that the display of a particularattribute or component to a particular viewer is made dependent uponcertain rules established by the content creator (e.g., the “price”attribute will be shown only to viewers over the age of 30, while allother viewers will be presented with the “third line of ad copy”attribute). In addition to such “intra-content” customization, incertain embodiments, more than one mapping between the media stream and(potentially different) items of supplemental content may be created.Such a plurality of mappings allows the media stream S to be displayed,for example, with a first set of supplemental content if embedded in ablog as opposed to with a second set of supplemental content if embeddedon a social networking web site. Later, as described below, theappropriate mapping M1 or M2 will be selected for provision to a viewerin accordance with the posting location of the annotated media stream.Further, different mappings may be provided for display on differentviewing devices (e.g., mapping M1 will be displayed to users viewing ona full-sized computer monitor, mapping M2 will be displayed to usersviewing on a portable device such as a telephone or personal digitalassistant, and mapping M3 will be displayed to users viewing on atelevision set). Further, different mappings may be provided for displaywith respect to specific viewer demographics (e.g., mapping M1 will bedisplayed to teenaged, female viewers, mapping M2 will be displayed toteenaged, male viewers, and mapping M3 will be displayed to viewers withany other demographic characteristics as a “default” mapping).

For example, a content creator who is mapping an advertising item ofsupplemental content featuring a diamond necklace may wish to present a“Treat Yourself!”-themed item of supplemental content to females and a“Show her you care!”-themed item of supplemental content to males. Tofacilitate the presentation of the relevant item of supplementalcontent, the content creator may designate a “set” of items ofsupplemental content that are intended to be presented in associationwith a specific temporal segment of the mapped media stream. This set ofitems of supplemental content may be assigned to another item ofsupplemental content that serves as a “placeholder” indicating that morethan one item of supplemental content has been or may be mapped to thesame segment identifier (identifying the same segment of the mediastream). Such placeholder designation may be by direct instruction(e.g., “drag-and-drop” of each of the relevant items of supplementalcontent into one “basket”) or indirect instruction (e.g., by mappingmultiple items of supplemental content to a common segment identifier).The content creator may then indicate the rules, instructions, and/orcriteria that will dictate which of the items of supplemental contentwill be presented in a specific context. As discussed above, these rulesmay rely on the specific characteristics of a viewer (e.g., age,gender), the technical/performance characteristics of the device onwhich the primary media stream is being displayed (e.g., a televisionversus a cellular phone), or the nature of the web site on which theannotated media stream is being displayed (e.g., a “kids-oriented” website versus a “sports-oriented” web site).

In addition, as also discussed above, designations of items ofsupplemental content may be made conditional on any number of factors.For example, a plurality of attributes (or a plurality of items ofsupplemental content for a particular segment of a media stream) may beprovided along with a choice regarding which specific attribute of anitem of supplemental content to present: (1) the number of times aviewer has viewed the media stream; (2) the number of times the mediastream has been viewed in total by all viewers or by a specific sub-setof viewers; (3) whether or not the viewer clicked on or otherwiseexpressed interest in a previously associated item of supplementalcontent in the mapping; (4) the average viewing time of the viewer (suchas, for example, if the viewer typically views only the first thirtyseconds of posted media streams); (5) the viewer's express or learnedpreferences with respect to factors such as the types of articles he orshe is interested in or the “style” of supplemental content (e.g.,advertising content) that he or she responds to best; (6) the actualamount of time that has passed since the item of supplemental contentwas first presented (such as with respect to an associated product thatis “on sale” for only a set number of days); and (7) a random choice.

All conditional factors for determining the selection of items ofsupplemental content may be placed in a hierarchy of significance orotherwise “weighted” so as to achieve a particular emphasis (such as,for example, presenting the same item of supplemental content to aviewer at least X times before other factors are considered andprocessed). These factors and the respective significant weightings maybe altered either in accordance with a set plan (potentially determinedby the content creator or by standards established by the accessprovider) or randomly (with the results being collected and analyzed tofacilitate the adjustments necessary to optimize the viewer's responseswith respect to specific criteria, such as the rate of viewerclick-throughs on a specific item of supplemental content). Multipledifferent combinations of weightings may be utilized (potentially asestablished by the user, by the access provider, or even automaticallyestablished by a system that has been programmed to apply differentweightings), in order to observe the impact upon viewer response (suchas measured by the rating that the viewer gives to the media stream orthe click-through rate of specific items of supplemental content), andto use the impact information to select optimal weightings and/oroptimal items of supplemental content to present to future viewers.

In one embodiment, items of supplemental content may be classified bythe time period in which the item of supplemental content should bedisplayed before being replaced in favor of another item of supplementalcontent. For example, a given item of supplemental content may be anadvertisement for a televised event such as a football game that is tobe broadcast on February 6^(th) at 6:30 pm EST. An “expiration” date maybe set for the advertisement to February 6^(th) at 5 pm EST, after whichtime the selected replacement item of supplemental content will bedisplayed to viewers (or, potentially, the advertisement will be removedentirely).

Further benefits to the access provider, to the content creator, and toadvertisers and sellers may be provided by enabling selection of ageneric product “type” item of supplemental content rather than aspecific item of supplemental content (e.g., a placeholder), for which aspecific item of supplemental content selected from a range of potentialitems of supplemental content would be selected and displayed to aviewer in a specific context. For example, a generic placeholder mayinclude only a particular product's UPC (Universal Product Code), GTIN(Global Trade Item Number), or an equivalent unique identifier for theproduct. Advertisers and sellers can then pay (directly, or through anauction method) to have a specific item of supplemental contentassociated with media streams mapped (via segment identifiers) toplaceholders that include that specific UPC, GTIN, or other uniqueidentifiers. For example, a user may choose to feature a Brand X MP3player in association with his/her media stream, with identifier N. Atan auction to purchase the association for identifier N for a particularperiod of time, sellers offer a range from $1 to $4 per transaction(where transaction could be views, clicks, or purchases). The selleroffering $4 is chosen by the auction, and for the particular period oftime, that seller's selected item of supplemental content (e.g.,advertising) will be included in the mapping.

In a more complex scheme, sellers could pay only for subsets of theproduct (e.g., identifier N for media streams viewed by a particulardemographic). In this way, a number of sellers could be potentiallyassociated with a particular media stream, but the display of any oneseller's advertisement is optimized to meet the relative value placed onsuch placement by the advertiser. This optimization process could becombined with other optimization processes so that the actual item ofsupplemental content that is displayed to a viewer is determined by theseller that has offered to pay the highest price for such display, butonly after taking into effect the relative click-through ratios asgenerated by previous viewers of the item of supplemental content. Forexample, Seller A may be willing to pay $1 per viewer click-through, andSeller B may be willing pay just $0.20 per click through. However, afterthe first 10,000 views of the video, the click-through rate generated bySeller B's item of supplemental content is more than five times greaterthan the click-through rate generated by Seller A's item of supplementalcontent. In such circumstances, the present invention may be optimizedby presenting Seller B's item of supplemental content only to subsequentviewers (after the first 10,000), because the per-click revenue is lower(but the aggregate revenue generated will likely be higher if similartrends continue). In another embodiment, attributes within a given itemof supplemental content may be allocated to sellers such that the itemof supplemental content that is displayed to viewers remains largely orentirely unchanged, but other attributes of the item of supplementalcontent may change according to the highest bid price. For example, itis generally the case that any given article is available for purchasefrom a number of different sellers or sources (e.g., a given DVD may beavailable from the filmmaker's website, from the movie company, fromvarious stores, from online sellers, or even from individuals such asthose who auction used merchandise utilizing online auction web sites),and although there may be only one item of supplemental content relatingto the article, any number of different sellers may be interested inbeing candidates for potential selection as the fulfillment source withrespect to a particular click-through relating to the article.

In embodiments where the use of a given item of supplemental contentinvolves a form of compensation (e.g., advertising revenue or e-commerceroyalty), the present invention may be configured to offer the option tohave the access provider automatically select the supplemental contentto be included in a mapping, in return for the access provider keeping alarger portion of the compensation. This may be of particular interestto content creators who wish to minimize the amount of additional effortneeded to monetize their media streams or, for example, to contentcreators who believe that the automated selections will be superior towhat they would select. This automated allocation of supplementalcontent could be entirely randomized (such as, for example, matching anitem of supplemental content that has been randomly selected from asupplemental content library and presenting the item of supplementalcontent to a viewer of the media stream for a random period of timeconsisting of between x and y seconds).

All factors considered in automated supplemental content selection canbe automatically utilized in the optimization schema, so that variouscombinations of or alternative items of supplemental content areproposed until the optimum combination is determined (e.g., byreferencing specific criteria). For example, the measuring criteria mayinclude the number of items of supplemental content that are “clicked”by viewers of the media stream, the average length of time that a givenmedia stream is viewed (e.g., few media streams are viewed tocompletion, and it may be the case the certain color combinations orsupplemental content display frequency engenders greater average viewingtimes), the average quality “rating” awarded by viewers of the mediastream or the volume of response (either words or number of submissions)generated by the annotated media stream. All of this information can beprocessed by an automated optimization system so that variouscombinations of items of supplemental content are proposed until therelevant measuring criteria are optimized.

Each of the optimization processes described above could be linked to agiven media stream so that new supplemental content mappings aregenerated each time the media stream is accessed (i.e., the mappings aregenerated in substantially real time). Alternatively, the optimizationprocesses could be set to run at various intervals so that, for example,x different versions of the mapping are generated (where each versioncontains supplemental content variations that will be analyzed foroptimization purposes), and a viewer request to view the media streamwill generate one of those mappings. When a predetermined number ofviews is completed, the optimization process will generate y differentversions of the mapping and repeat the process. This latter approachwould avoid the necessity to provide near-real time processing, butmight increase the storage costs of storing multiple mappings for thesame media stream.

In one embodiment, users are given the option to personally classifysupplemental content according to one or more characteristics, such asby classifying the supplemental content as “advertising” (e.g., possiblyincluding a purchase opportunity), “content complementary” (e.g.,graphic images that complement the media stream), “navigation” (e.g.,supplemental content in the form of a video that allows the viewer toeasily navigate to another part of the media stream or to another mediastream) or the like. This classification can be performed eitherautomatically, by reference to certain criteria such as the by type ofinformation or components that are included in the supplemental content(e.g., the inclusion of an item of supplemental content in the“Retail_Price” category of a library entry might indicate that thesupplemental content is “advertising”), or manually such as by anemployee or user of the access provider.

Such classification may also enable any number of additional processesthat provide users (e.g., content creators) with additional control overthe media stream and the access of viewers to the media stream. Forexample, the user may provide authorization for viewers to providealternative supplemental content for a given category of supplementalcontent (e.g., viewers may suggest alternatives that include onlyadvertisements that were created by the content creator and that havenot yet produced X level of viewer response), while at the same time“locking” supplemental content of all other classifications (e.g.,viewers cannot suggest alternatives to “artistic statements” created bythe content creator as part of the overall multimedia experience). Thiswould facilitate, for example, preserving the presentation of anartistically annotated media stream despite a potential tendency toincrease the number of advertising-oriented items of supplementalcontent associated therewith as the media stream increases inpopularity. It may also be possible, for example, for the user to, withrespect to a given media stream, “open” or “lock” the association ofspecific types of supplemental content, such as blocking the ability forcollaborators to provide supplemental content that is presented prior tothe commencement of presentation of the primary media stream, whileexpressly encouraging the creation of supplemental content that allowsthe viewer to easily access a related media stream selected by theoriginal content creator.

As another example, a user (e.g., a content creator) may provide anannotated media stream for which the first item of mapped supplementalcontent is an artistic (i.e., non-commercial) item of supplementalcontent that is displayed from time t=0 seconds to time t=10 seconds,the second item of supplemental content is an advertisement that isdisplayed from time t=10 seconds to time t=20 seconds, and the thirditem of supplemental content is a navigation assistant that is displayedfrom time t=20 seconds to time t=30 seconds. User A could utilize theauthorization system to “open” only the advertisement (i.e., the seconditem of supplemental content) to syndication, so as to maximizepotential revenue derived from the associated item of supplementalcontent, while at the same “locking” all other supplemental contentcategories, so as to ensure the preservation of the core viewerexperience that User A seeks to create with the first and third items ofsupplemental content. This functionality would, for example, allow theuser to “lock” the advertising that is meaningful to him/her (e.g., anadvertisement for a specific snowboard that the user is riding in themedia stream), while allowing others to provide advertising that isappropriate to other, more generalized aspects of the media stream(e.g., an advertisement for ski jackets that may or may not look similarto a jacket depicted in the media stream).

If the user has created a new item of supplemental content, the item ofsupplemental content can be uploaded or “authorized” to be includedwithin the library so that when others view the library, they may alsobe presented with all or some of the individual items of supplementalcontent therein that have been derived from specific library entries.The user may provide the item of supplemental content subject to variousconditions, such as allowing others who visit the relevant library entryto use the item of supplemental content “as is” (i.e., “locked”) or,alternatively, allowing the item of supplemental content to be used forany purposes (including as a template for another user to modify and/orcombine with additional items of supplemental content). Alternatively,items of supplemental content that meet certain criteria may beautomatically included within the library (e.g., all items ofsupplemental content that generated click-through rates in excess of x %are placed within the library). This process of automatically includingitems of supplemental content may result in items of supplementalcontent being placed in a “holding” area of the library, in which theitems of supplemental content are identified as items of supplementalcontent that have yet to be associated with a specific library entry.These items of supplemental content may be stored in the library with aparticular library entry, potentially on a provisional basis thatrequires confirmation by other users or by an administrator. When anitem of supplemental content is identified in this regard, anotification may be provided to the creator of the item of supplementalcontent. If the user fails to “claim” the item of supplemental contentwithin a specific period of time, the access provider may label the itemof supplemental content as an “access provider” item of supplementalcontent (for which the access provider receives the “creator” portion ofany generated revenue), as a “community” item of supplemental content(for which no revenue is claimed or, alternatively, the creator portionof the revenue is placed in a specific account perhaps distributed tothe community as a whole on a pro rata basis, donated to a charity, orsome other purpose) or the like.

In one embodiment, this same type of approach that may be utilized to“open” or “lock” certain supplemental content categories (e.g.,syndicators allowed only to change the advertisements) may be applied tospecific components within a given item of supplemental content. Forexample, a user may “open” the ability for others to provide alternative“styles” of the item of supplemental content (e.g., allow the backgroundcolor and fonts to be changed), but “lock” other attributes (e.g., theproduct image, retail price, and actual ad copy). This samefunctionality may be applied to specific library entries.

In one embodiment, items of supplemental content may be created andadministered by an individual using tools provided by or hosted by theaccess provider. One such tool may include a set of procedures thatallows a user to upload some individual components (e.g., fonts, images,etc.) of an item of supplemental content, select some individualcomponents of supplemental content from the supplemental contentlibrary, transform one or more of the uploaded and/or selectedindividual components (e.g., by changing the colors, adding soundeffects, etc.), and compile the customized individual components into acomposite item of supplemental content that is formatted forcompatibility with the media stream and with the technical needs of theaccess provider. This process may be conducted within a portion of theaccess provider that is made available to the user for his/her orhis/her organization's personal use (e.g., a private “workspace”).Alternatively, the access provider may provide a public posting of“in-progress” items of supplemental content, whereby the tool allowscertain approved groups or individuals (or even the general public) tocontribute to the creation of items of supplemental content by offeringcommentary, additional individual components, modifying the in-progressitem of supplemental content, offering alternative items of supplementalcontent, etc. It may be the case that a user would utilize a privateworkspace to create an alternative item of supplemental content, providethe alternative item of supplemental content to the public workspace forcommentary and modification by the community, and then return thealternative item of supplemental content to his/her private workspacefor final modifications. In one embodiment, this workspace tool enablesa “revision history” function, so that the user and/or others can trackthe progress of an item of supplemental content's development, accessprior instantiations, determine who made particular changes and when,etc.

Another tool may allow a user (e.g., a content creator) to designate aset of items of supplemental content as the “working library” of theitems of supplemental content from which the user will create theannotated media stream (i.e., the combination of the media stream, themedia stream player application, and the collection of items ofsupplemental content for the mapping). In one embodiment, the workinglibrary may be modified at any time, saved for access at a later date,shared with others, etc. in a manner similar to the workspace.

In one embodiment, the working library includes functionality such thatif a user would like for more than one item of supplemental content tobe available for mapping with respect to a specific temporal segment ofthe media stream (e.g., a first item of supplemental content will beshown to male viewers and a second item of supplemental content will beshown to female viewers and viewers of unknown gender), the user mayutilize a “proxy” or “placeholder” to which multiple items ofsupplemental content may be assigned. The tool may allow the user toplace the items of supplemental content in an order in which the useranticipates that the items of supplemental content will be displayed toa viewer (i.e., when the items of supplemental content are ultimatelymapped to segment identifiers associated with the media stream), and toeasily “mix-and-match” various combinations of items of supplementalcontent to determine the most suitable order, potentially via“drag-and-drop,” numbering systems, or other methods for assigning theitems of supplemental content to a sequence.

In one embodiment, users (e.g., content creators) are presented withsimple menu selections to determine their interest level in providingitems of supplemental content. For example, as illustrated in FIG. 40,which illustrates an exemplary initial display 4000 for presentation toa user, the user may be presented with yes or no questions such as“Would you like to add supplemental content to your video?” and/or“Would you like to allow others to add supplemental content to yourvideo?” A “default” answer could be provided for these questions, sothat users that are not interested in annotating their media streamsneed only take minimal additional steps. Providing a system that is easyto bypass may be very important, because a complicated system mightdeter all but the most dedicated users from contributing media streams.

In the course of creating items of supplemental content and mappings, auser may benefit from being able to keep track of the items ofsupplemental content that have been created and the items ofsupplemental content that may still be required. To facilitate this, thesystem may provide the user with the ability to make a list and/or notesregarding specific items of supplemental content, where individuallistings and notes are severable into various workspaces or portions ofa workspace. For example, one user may wish to create a list of all ofthe items of supplemental content that are envisioned for a particularmedia stream mapping, and then transmit the entire list to a group ofpotential contributors (potentially allocating responsibility for eachitem of supplemental content on the list to a specific member or membersof the group). Alternatively, the system may provide the user with theability to create “placeholders” for the items of supplemental contentthat he/she intends to create. For example, a user who is creating itemsof supplemental content for a mapping associated with a fishing-themedvideo may wish to create placeholders or proxies such as “title screen,”“picture of boat,” “map of the lake,” “advertisement for fishing pole,”“animation of fish towing a boat,” “bait advertisement,” “photograph ofthe fish dinner that night” and “closing screen.” The present inventionwould then allow the user to create items of supplemental content andrelate them (e.g., by “drag-and-drop,” by numbering, etc.) to each ofthe placeholders. The present invention may also allow the user torelate multiple items of supplemental content to the same placeholder,so as to facilitate the content creator's ability to easily comparedifferent versions of the same item of supplemental content. Thecollection of placeholders and items of supplemental content thus formsa “working library” that will facilitate the user's interactions withthe items of supplemental content, either prior to or after the initial“release” of the annotated media stream for viewing. The system may alsopermit the user to designate a specific order of appearance for theplaceholders (e.g., via “drag-and-drop,” numbering systems, or otherwiseassigning the placeholders to a sequence). In one embodiment, the systemallows the content creator to take various actions with respect to theselected items of supplemental content and/or placeholders such as toadd, delete, modify, rename or re-order such placeholders.

The method 700 thereby allows a user (e.g., a content creator) to createa mapping from segment identifiers associated with a media stream to oneor more items of supplemental content. In one embodiment, mappings aremade between segment identifiers identifying temporal segments of themedia stream and one or more items of supplemental content (e.g., thefirst minute of the media stream may be associated with the first itemof supplemental content, the second minute of the media streamassociated with the second item of supplemental content and so on). Inanother embodiment, mappings between the segment identifiers and itemsof supplemental content are made with respect to an event or action suchas when the viewer has “paused,” or “fast-forwarded,” or “stopped” themedia stream, the amount of time that has elapsed since the “play”button on the media stream player was pressed, or upon completion of themedia stream upload to the viewer's display device. For example, theuser may configure the mapping so that when the media stream is paused,the viewer is presented with a specific item of supplemental content,regardless of the subject matter appearing in the particular scene ofthe media stream at which the media stream was paused. In anotherembodiment, mappings to items of supplemental content may be made withrespect to metadata included in the encoding in the primary media streamor added to the encoding by the user or by the access provider (e.g.,segment identifiers, cue points and the like).

Alternatively, the present invention may reference all items ofsupplemental content in the working library created by a user andautomatically apportion the items of supplemental content to the segmentidentifiers in accordance with a default apportionment (e.g., each itemof supplemental content is associated with a temporal segment equal tothe total time of the media stream divided by the number of items ofsupplemental content being mapped). In one embodiment, the presentinvention allows the user to adjust the default mapping so that eachitem of supplemental content is associated precisely with those temporalor other segments (e.g., by “scene” or in a synchronization that createsa meaningful connection between the media stream and a given item ofsupplemental content) of the media stream, as desired by the user. Thismapping adjustment process may be performed by various means such as bythe input of segment identifiers (e.g., supplemental content item number1 is mapped to the segment identifier identifying the segment that spanstime t=1 second to time t=17 seconds) or by adjusting “bars” or othergraphical user interface elements along a time line representing theentire length of the media stream (such that a first bar indicates thepresentation of a first item(s) of supplemental content, a space betweenthe first bar and a second bar represents the time in which the firstitem(s) of supplemental content will be presented to a viewer, and thesecond bar represents when the first item(s) of supplemental contentwill be replaced by a second item(s) of supplemental content).

Alternatively, items of supplemental content may be mapped to segmentidentifiers that are independent of a particular media stream. Forexample, a selection of items of supplemental content may be mapped toappear every fifteen seconds, with the total number of items ofsupplemental content presented to the viewer being determined solely bythe length of the media stream with which the items of supplementalcontent are associated (e.g., if a mapping consists of a total of tenitems of supplemental content, but the media stream associated with themapping is only seventeen seconds long, only the first two items ofsupplemental content will be presented to viewers). Alternatively, allitems of supplemental content that were mapped beyond the length of themedia stream may continue to be displayed after the media streamconcludes (i.e., the items of supplemental content continue to bepresented, perhaps at a quicker pace as set by the user for post-mediastream display).

In some embodiments, the method 700 may provide a tool whereby each itemof supplemental content in a mapping is numbered from 1 to n. The usermay associate the items of supplemental content with the media stream bysimply watching the media stream and pressing the number(s) associatedwith the item(s) of supplemental content that should be mapped to thesegment identifier(s) identifying that point. The association of theselected item(s) of supplemental content with the media stream would endwhen the next item of supplemental content is selected, thereby creatinga temporal association from the start point to the end point with theselected supplemental content (i.e., the start point of the latter itemof supplemental content necessarily defining the end point of theimmediately preceding item of supplemental content).

Referring back to FIG. 7, in step 708, the method 700 synchronizes thesupplemental content with the media stream, to produce an annotatedmedia stream. Specifically, as discussed above, the method 700 maps thesupplemental content to appropriate segment identifier, as indicated bythe user. In one embodiment, the method 700 maps at least one segmentidentifier in the media stream to multiple items of supplementalcontent.

In optional step 710 (illustrated in phantom), the method 700synchronizes system-selected supplemental content with the media stream.For example, if the supplemental content involves a form of compensation(e.g., advertising revenue or e-commerce royalties), the method 700 maybe configured to automatically select specific articles to be mapped tothe segment identifiers in the media stream, for example in exchange forkeeping a larger portion of the compensation. Alternatively, advertisingspace may be bought from the user (i.e., the provider of the mediastream) and resold to third parties (e.g., advertisers). In thisembodiment, advertising space may be sold using a price list or anauction system.

The method 700 saves the annotated media stream in step 712 (e.g., uponindication from the user that annotation is complete), beforeterminating in step 714. In one embodiment, saving the media streamincludes saving user-provided tags associated with the media stream(e.g., that describe the media stream, assign the media stream to agenre, or specify certain characteristics of the media stream such asthe length, the content creator, the director, or the like). In oneembodiment, saving the media stream includes encoding the media stream.In one embodiment, encoding is performed in a manner that allows both aprogram identifier (i.e., identifying the particular media stream) and asegment identifier associated with each segment of the media stream tobe obtained. In a further embodiment, the saved annotated media streamis associated with one or more saved parameters, such as a cap on“click-throughs”. This may be useful, for example, where a sellercreates an annotated media stream for a specialized product. Forexample, if a seller cannot profitably sell more than one thousand unitsof a given article, the seller may not want to pay for full advertisingonce a certain number of click-throughs has been reached. In oneembodiment, once the capped number of click-throughs is reached, theannotated content reverts to a more “bare bones” structure (e.g., justmanufacturer and item name, but no purchase information or enablement).

Once the annotated media stream has been saved, it may be made availableto viewers (for example, by publishing the annotated media stream on aweb site or by providing the annotated media stream to an accessprovider). The method 700 therefore allows a user, such as a contentcreator, an ad buyer or a seller, to control the annotation of a mediastream. Thus, the method 700 may function as a portal that provides auser with a means of creating annotated media streams (e.g., in the caseof content creators) and of providing articles for use in media streams(e.g., in the case of sellers and/or ad buyers). In one embodiment, themethod 700 executes in accordance with a local application on the user'scomputer. In another embodiment, the method 700 executes in accordancewith a web-based application. In yet another embodiment, portions of themethod 700 may be performed locally, while other portions of the method700 are performed remotely via a networked computer.

Further embodiments of the method 700 enable outsourcing of the mappingtasks. That is, the party who uploads or provides the received mediastream may not be the same party that performs the mapping tasks (e.g.,such as selection of supplemental content). In one embodiment, the mediastream may be received in a protected (e.g., encoded) form, or may beprotected upon receipt, so that further steps of the method 700 (e.g.,mapping of supplemental content, etc.) can be outsourced in a securemanner. In one embodiment, the protection prevents remote downloading ofthe media stream (i.e., the media stream can only be viewed). In anotherembodiment, the media stream is received in discrete “scenes” orsections to ensure that no individual performing subsequent mappingtasks is able to view more than one scene of the media stream. In thisembodiment, an individual performing mapping tasks with regard to agiven scene may leave “notes” for individuals performing mapping taskswith regard to other scenes of the same media stream, in order to informthe other individuals of mapping decisions that have been made.

In a further embodiment, a log in/log out system allows individuals toperform mapping tasks with regard to a given media stream. In oneembodiment, this log in/log out system illustrates changes made during aprior log in session.

In one embodiment, selection of supplemental content for mappingpurposes is optimized by enabling the access provider to access therecords of an existing digital video recorder, video-on-demand service,or other type of “cache” in order to alter supplemental content alreadyassociated with media streams stored therein or already viewed by theviewer. Using the segment identifiers, it can be determined which theitems of supplemental content did not produce a viewer response, andedits can be made to the mapping with relation to these segmentidentifiers. In this way, the supplemental content displayed inconnection with a media stream might actually change from one viewing toanother.

In one embodiment, viewers are prompted to provide additionalinformation regarding their preferences with respect to supplementalcontent. This viewer information could be requested when, for example, aset top box is first hooked up, or at any time thereafter by the vieweraccessing an online service account. The indicated viewer preferencescould be matched with the viewer's actual behavior in order to determineif the viewer preferences provide sufficiently accurate data for thepurposes of selecting supplemental content. If the preferencessubstantially match the behavior, the indicated viewer preferences areconsidered to be of very high value (in that data-mining predictionswill have been, in essence, certified by the viewer's own actions andconscious preferences). This information could be used enhance thedigital placement of particular articles into actual media streams(e.g., Brand X beer, not Brand Y beer), to enhance the use of specifictypes of music or other audio components (e.g., Hip Hop), to enhance theuse of specific types of advertising (e.g., more “information rich” asopposed to “what your friends/viewers like you” are buying) and thelike.

In one embodiment, viewer behavior could be guided by requiring viewersto “link” various tokens obtained through different actions. Forexample, a “game” could be implemented that takes advantage of variousforms of video tokens (e.g., to get a discount for a given article, theviewer must collect tokens by watching each of three specific movies ata given movie theater, by watching a television show that features aspecific brand of automobile, and by purchasing a particular article ata particular seller). In one embodiment, tokens must be collected in aspecific order (e.g., in the example above, the viewer is required tofirst see the television show and then see the movies). In oneembodiment, token functionality is utilized as a “scavenger hunt” inwhich the individual media stream and/or supplemental content associatedtherewith does not directly participate, but is nonetheless able tofacilitate the search. For example, each item of supplemental contentmay be placed with an ontology. Therefore, a scavenger hunt couldrequire the viewer to obtain one token from any television showfeaturing an article of category X of the ontology and one token fromany theatrical release including a category Y article.

Sometimes, a particular group of viewers may be interested in receivingcustomized supplemental content with respect to a given media stream.Thus group could be provided with a mechanism to establish “private”supplemental content that is accessible only to the group. For example,there are many distance learning applications in which a school mightsend a video about a particular author out to all of its students. Inone embodiment, this feature could be combined with a mechanism fortracking viewer response (e.g., a mechanism that verifies that eachstudent clicked through all supplemental content so as to verify thathe/she watched the entire video). As another example, individuals cancreate a customized annotation about a media stream segment, such asranking the segment (e.g., 4.5 out of 5 stars) or making notes about thesegment (e.g., “Sarah—that dress would look great on you”).

It is increasingly the case that advertisements are being inserted intolive media streams. For example, the area behind the batter in baseballbroadcasts is often digitally edited to show different advertisementsevery few minutes. To facilitate the presentation of appropriatesupplemental content with such media stream, a temporary mapping may becreated so that a given segment references information that is specificto the “raw” segment, but also that is specific to what has beenpresented to a particular viewer.

FIG. 14 is a flow diagram illustrating another embodiment of a method1400 for annotating media streams, according to the present invention.The method 1400 may be implemented, for example, in its entirety by thestream annotation service provider 112 of FIG. 1. Alternatively, themethod 1400 may be implemented by the stream annotation service provider112 in conjunction with a content creator 102.

The method 1400 is initialized at step 1402 and proceeds to step 1404,where the method 1400 obtains a media stream. As described above, themedia stream comprises at least one of: a video signal, an audio signalor a text signal, for distribution over a user device.

In step 1406, the method 1400 logically divides the media stream into aplurality of segments, wherein segment identified by a segmentidentifier (e.g., a unique segment identifier). As described above,these segments identifiers may comprise, for example, a segment numberor set of segment numbers, a generated number, time-codes or otheridentifier. In one embodiment, the segment identifiers are embedded inthe media stream.

In step 1408, the method 1400 maps at least one of the segmentidentifiers to at least one intermediate node representing an article, aconcept or a class of concepts. This creates a first mapping. Thus, forexample, the first mapping (to the intermediate node) may represent thatthe segment of the media stream identified by the segment identifiercontains, suggests, implies or in some other way conveys the article,concept or class of concepts represented by the intermediate node. Inturn, intermediate nodes are independently mapped via a second mappingto specific items of supplemental content. Thus, a segment identifierthat is mapped to an intermediate node is also indirectly mapped (viathe intermediate node) to supplemental content.

The mapping is stored in step 1410. In one embodiment, the mapping isstored in a store that is indexed according to a unique identifier forthe media stream. The method 1400 then terminates in step 1412.

The use of the intermediate nodes in accordance with the method 1400provides many advantages. For example, the use of intermediate nodesmakes the remapping of supplemental content very efficient. To change anitem of supplemental content (e.g., if the item of supplemental contentbecomes unavailable), only one mapping (i.e., from the intermediate nodeto the item of supplemental content) need be changed, and this changewill map the new item of supplemental content to all segment identifiersthat are mapped to the intermediate node. In situations where thesupplemental content appears, say, thousands of times in hundreds ofmedia streams, changing one mapping scales much more easily thanchanging thousands of mappings in each of the hundreds of media streams.

Additionally, because the specific items of supplemental content are notembedded in or with a media stream, the items of supplemental content donot need to be identified prior to any particular event such as thecommencement of filming or encoding of the media stream. This allows theentire creation process to be much more flexible. For example, even ifan item of supplemental content is unknown at encoding time, a segmentidentifier can be mapped to an intermediate node. If an item ofsupplemental content is later identified, the item of supplementalcontent can be mapped to the intermediate node and thus automatically(indirectly) mapped to all segment identifiers that are mapped to theintermediate node.

Furthermore, one can easily identify mappings from segment identifiersthat do not have (indirectly) mapped supplemental content. As theapproach embodied in the method 1400 allows for “half” mappings (i.e., afirst mapping of a segment identifier to an intermediate node without asecond mapping of the intermediate node to an item of supplementalcontent), one can easily search for intermediate nodes that are notmapped to items of supplemental content. This enables searching for“unused” or “unsold” intermediate nodes, and the straightforwardsubsequent addition of an item of supplemental content.

Consistency is important in order to most effectively utilize theintermediate nodes across a plurality of segment identifiers associatedwith a plurality of media streams. For example, if a first media streamis encoded to map segment identifiers identifying romantic segments tothe intermediate node “romance”, but a second media stream is encoded tomap such segment identifiers to the intermediate node “love”, some ofthe advantages of the method 1400 may not be fully realized. Hence, acarefully controlled vocabulary is preferentially used to constrain thefirst mapping of segment identifiers to intermediate nodes.

A constrained vocabulary will give rise to issues with homonyms. Forexample, is the intermediate node “glasses” related to a plurality ofdrinking glasses, or to a pair of eye glasses? To overcome this problem,the intermediate nodes in one embodiment are considered to be instancesand classes of an ontology. The ontology provides a universe of possibleintermediate nodes that are classes (abstract groups, sets orcollections of objects) and instances (both concrete objects such asspecific cars, and abstract objects such as numbers, words and themes).Instances are members of a class (e.g., a “Brand X four-wheel drivetruck” is an instance of the class “truck”). In one embodiment, classesare arranged in a graph, providing at least an “is-a” (i.e., an instance“is-a” member of the class, and a class “is-a” member of thesuper-class). Such an “is-a” relationship thereby creates a hierarchicaltaxonomy. Each object (class or instance) in the ontology may have anumber of attributes associated with it. For example, the class“business” may have an attribute of “location”, as all business have alocation.

In some embodiments, a “part-of” relationship, sometimes known as ameronymy relationship, is provided between classes (or between instancesand classes), representing how objects (or classes of objects) combinetogether to form a composite object. For example, the instances “Jelly”and “Peanut Butter” are both “part-of” the instance “Peanut Butter andJelly Sandwich”. The ontology structure allows more than one parentnode, under both the “is-a” and the “part-of” relationships, creating adirected acyclic graph of instances and classes.

FIG. 8 is a flow diagram illustrating one embodiment of a method 800 forproviding annotated media streams to viewers, according to the presentinvention. The method 800 may be implemented, for example, in itsentirety by the access provider 106 of FIG. 1 (e.g., via a user devicesuch as a television, a DVD player, a DVR, a set-top box, a portablemedia player, a personal digital assistant, a personal computer, amobile phone, an in-flight or in-vehicle entertainment system, a videogame console, a dedicated device or the like). Alternatively, the method800 may be implemented by access provider 106 in conjunction with thestream annotation service provider 112.

The method 800 is initialized at step 802 and proceeds to step 804,where the method 800 receives a mapping of segment identifiersassociated with a media stream to items of supplemental content. In oneembodiment, the supplemental content comprises commercial informationrelating to one or more articles appearing in the media stream, asdescribed above. In one embodiment, the mapping is received from aremote server (e.g., the stream annotation service provider 112), forexample, via the Internet, via a cable network (e.g., where the remoteserver may be the cable head-end, or the head-end may mediatecommunications with the remote server) or via satellite. Particularembodiments may receive the mapping as an OpenCable application platform(OCAP) or digital video broadcasting (DVB) application, or as a privatesection with an MPEG transport stream. In a further embodiment, themapping is received “on the fly” from a “director” (e.g., contentcreator 102 or stream annotation service provider 112) who creates alist of supplemental content relating to articles that may be relevantto the media stream, and, as the media stream is distributed ordisplayed, “activates” specific items of supplemental content from thelist in substantially real time as the media stream conveys informationabout the related articles. In one embodiment, the mapping furtherincludes the URL of one or more World Wide Web sites relating to one ormore articles appearing in the media stream.

In step 806, the method 800 displays the media stream via a displaydevice. In one embodiment, displaying the media stream includespresenting an indicator when the segment identifier identifying thecurrently displayed portion of the media stream (e.g., the currentlydisplayed segment) is mapped to supplemental content. Thus, a viewer ofthe media stream would automatically know when additional informationabout one or more articles being displayed is available (e.g., asopposed to having to guess or query). In one embodiment, the indicatoris presented when the segment identifier identifying the currentlydisplayed portion of the media stream is mapped to supplemental contentrelating to information that is expected to be of interest to the viewer(e.g., based on demographic information, stated viewer preferences orprevious viewer activity). In one embodiment, the indicator is a visualindicator, such as an icon overlayed on the display of the media streamor a (perhaps flashing) light on the media stream player (e.g., aset-top box, DVR or the like), on the display device or on a remotecontrol. In further embodiments, the indicator may be an audibleindicator or other sensory indicator (e.g., a vibrating remote control).In a further embodiment still, a “split screen” display may be employedthat displays the media stream in a first portion of the display anddata relating to currently accessible supplemental content in a secondportion of the display.

The method 800 then proceeds to step 808 and receives a first signalfrom a viewer of the media stream. The first signal indicates that theviewer wishes to view at least a subset of the information embodied inthe supplemental content. In one embodiment, the first signal isconveyed via the push of a button (e.g., on a remote control, akeyboard, a mouse, a light pen or a laser pointer) or via a spokencommand (e.g., to a microphone, a mobile phone or a voice-over-IPhandset). For example, FIG. 26 illustrates an exemplary set-top box 2600that interacts with user signals provided via a remote control 2602. Inone embodiment, the method 800 supports multimodal viewer interaction,meaning that two or more input modalities may be employed for receivingviewer signals. Multimodal interaction may be helpful, for example, incorrecting errors in processing viewer signals. For example, errors inprocessing viewer signals may be corrected in accordance with themethods described in U.S. Pat. No. 6,742,021, which is hereinincorporated by reference in its entirety.

In one embodiment, the first signal indicates that the viewer wishes toview the supplemental content immediately. In another embodiment, thefirst signal indicates that the viewer wishes to view the supplementalcontent at a later time (e.g., at the conclusion of the media stream).In further embodiments, the first signal indicates that the user wishesto save the present position in the media stream to view thesupplemental content at a later time, without specifically identifyingthe article of interest (e.g., “mark for later”). For example, FIG. 31illustrates an exemplary system in which a viewer signal (received,e.g., via a remote control 3100 to a set-top box 3102) indicates thatthe viewer wishes to “bookmark” supplemental content mapped to thecurrent segment's segment identifier for later viewing. As illustrates,the set-top box retrieves the relevant segment identifier and theviewer's viewer identifier. The supplemental content mapped to thesegment identifier is then retrieved from a database 3104 and sent to anaccount 3106 associated with the viewer identifier for later viewing.

Thus, in one embodiment, mappings or items of supplemental contenttherein may be “saved” (either automatically or on the viewer'sinitiative) for later viewing, either on the same display device onwhich the media stream is shown (e.g., after the media stream is played,the items of supplemental content are presented) or in on a differentdisplay device (e.g., sent to the viewer's account with an accessprovider or a third party). This may be particularly useful if the mediastream is viewed on a display that is too small to display the items ofsupplemental content effectively or in their optimal size (e.g., suchthat advertising text is illegible for the average viewer), such as whenthe media stream is viewed on a cellular phone. If sent to another typeof display device such as a television, an access provider may providefunctionality for the items of supplemental content to be separated fromthe media stream and directed in another way, such as being sent to aviewer's email account for later viewing, presented as a “split screen”or “picture-in-picture” or in some other manner that is suitable for thedisplay device.

In one embodiment, the functionality described above may be applied onlyto particular items of supplemental content. For example, the viewer maybe permitted to block the presentation of all items of supplementalcontent featuring “adult” content or all items of supplemental contentthat include advertisements mapped segment identifiers identifying morethan a specified percentage of the media stream's length. Thisfunctionality would allow the viewer to tailor his/her viewingexperience to meet his/her preferences. An access provider could chooseto make all such “blocking” features available solely to those viewerswho met certain requirements (such as having paid a certain fee). Theaccess provider may permit the establishment of a specific viewingaccount for which access privileges are defined by one viewer for allother viewers utilizing the account (e.g., a parent establishing anaccount on behalf of his/her child; a company establishing accounts onbehalf of its employees so that the employees may view onlycompany-provided media streams).

In step 810, the method 800 determines, in accordance with the mapping(e.g., mapping 2604 of FIG. 26), which segment identifier corresponds tothe point in time at which the first signal was received (i.e., whatsegment of the media stream was being displayed when the viewer sent thefirst signal). In one embodiment (e.g., where segment identifierscomprise hash-generated identifiers), this determination is made bycomputing the hash of the current frame of the media stream, identifyingthe time associated with the computed hash, and retrieving thesupplemental content for that time. In another embodiment, the time-codedata from the current segment is used. In another embodiment, thisdetermination accounts for a built-in time delay, as discussed above.For example, the first signal may be received at time t₀, but the viewermay be reacting to an article appearing in the media stream at timet_(0-n) (i.e., the viewer's reaction time is slower than the progressionof the media stream).

In another embodiment, data describing the actual segment (e.g., ascreen capture image of the specific segment that was being displayedwhen the viewer signal was received) is received by the method 800,potentially along with additional information (either from an embeddedprogram identifier or from the access provider) regarding the mediastream being viewed and/or any available timing information (e.g., thissegment was derived from x seconds into the media stream Y). The screencapture image could then be processed (e.g., “hashed”) to determine thesegment's unique digital signature (e.g., average luminance and color).This information can then be matched against a database of media streaminformation that had previously been: (a) encoded in each segmentidentifier; and (b) hashed in each segment to determine the uniquesignature. The hash from the viewer is then compared to the hashesavailable from the digital media stream (guided by the approximatetiming information). This could occur very rapidly due to the fact thatthe set of hashes to compare would be relatively small given the timecode information. The advantages of this approach are that: (a) nothingneeds to be included in the media stream itself; (b) the content creatordoes not have to encode anything (all he/she need do is provide themedia stream for encoding; and (c) the media stream itself does notinclude any data that could be misappropriated by others (e.g., usingthe data for unauthorized purposes).

The closed-captioning-related Extended Data Service (XDS, e.g.,Electronic Industries Alliance (EIA)-708) data channel that is used toprovide closed captioning operates by displaying particular sets ofwords beginning at particular points in a sequence of media streamsegments. The initial segment containing this information (e.g., thefirst segment to which a particular combination of words is associated)can be utilized as a relatively fixed reference point within the mediastream, and this is true regardless of segments that may have beenedited out. In one embodiment, this closed captioning information isused in conjunction with an “inertial” location method described in moredetail below in order to determine what supplemental content to provide.“‘Inertial’ location” refers to the fact that once a fixed point in amedia stream is identified, a “stopwatch” function can be used todetermine the specific segment in which a viewer signifies an interestat a later time. For example, the point in time occurring x secondsafter the first segment in which the phrase “Save me, Superhero” appearscan be equated to a specific segment identifier (or a relatively fewnumber of segment identifiers) which, in turn, then map to specificitems of supplemental content. In one embodiment, instead of “reading”the closed captioned transcript, voice recognition software could beused to recognize the speech. The recognized speech is then matchedagainst a transcript to determine where in the media stream to start theinertial positioning. One significant advantage of this approach is thatthe media stream that is displayed to a viewer does not require specialcoding. Rather, aspects of the media stream itself are utilized todetermine placement. In one embodiment, this information is matchedagainst a version of the media stream that is encoded, so as to enableeasier mapping.

In one embodiment, parameters are set for a relevant device (e.g., a settop box) to “read” the segment identifier(s) occurring at a precisemoment in time or under a precise set of conditions or viewer criteria.In one embodiment, this is implemented by sending a “pull” instructionor signal to the relevant device with the instruction to transmit thesegment identifier. The pull instruction could be sent out in accordancewith a variety of parameters, including any one or more of thefollowing: pulling segment identifiers from a national “statisticallysignificant” sample in order to determine nationwide viewing habits;pulling segment identifiers from a specific demographic subset (e.g.,determine what all viewers in towns of fewer than 10,000 people arewatching); pulling segment identifiers from a specific subset of viewersover a period of time, in order to track the viewing habits of thoseviewers (potentially using the viewing habits to create new sub-sets ofviewers, so that better sub-categories can be generated for the purposesof data mining); pulling segment identifiers from those viewers thathave watched a given source of media streams in the prior Y weeks;pulling segment identifiers from those viewers that clicked-though on aparticular item of supplemental content; pulling segment identifiersfrom those viewers whose profiles indicate that a given source of mediastreams is actually available to them through their access provider;pulling segment identifiers from those viewers currently watchingpay-per-view or video on demand; pulling segment identifiers from thoseviewers who are currently logged onto the Internet; or pulling segmentidentifiers every x seconds for all viewers watching a particular mediastream in order to determine when the viewers stopped watching the mediastream.

In one embodiment, the segment identifiers that are pulled (or “polled”)in a given context are stored so that the segment identifiers areaccessible in bulk and not in substantially real time. For example,every time a viewer changes media streams that he/she is viewing, it maybe helpful to know the media stream segment that appeared just prior tothe channel change (i.e., that the viewer had been watching segment X ofmedia stream Y before switching to media stream Z), as this may help todetermine the factors that cause the viewer to stop viewing a givenmedia stream.

The method 800 then proceeds to step 812 and displays, at theappropriate time (e.g., based on whether the user wishes to view thesupplemental content immediately or at a later time), the supplementalcontent that is mapped to the corresponding segment identifier. In oneembodiment (e.g., where the segment identifier is mapped to supplementalcontent relating to multiple items), the information is displayed in theform of a menu of selectable articles. For example, FIG. 27 illustratesthe system of FIG. 26 after the supplemental content 2700 mapped to thesegment identifier has been retrieved and displayed to the viewer viathe display device 2702. As illustrated, the supplemental content 2700is displayed as a menu including information for a purse, a sweater anda pair of sunglasses.

Thus, for instance, if the first signal was received at a time when oneor more segments identified by the segment identifier depicted anactress, the supplemental content mapped to that segment identifiermight include the actress's filmography and information about a sweaterthat the actress was wearing. Thus, the supplemental content displayedin response to the first signal might include an image of the actressand an image of the sweater, where each of the images comprises ahyperlink to World Wide Web page containing additional information. Inone embodiment, the display of the media stream is paused while thesupplemental content is displayed. In one embodiment, the supplementalcontent is displayed via the same display device as the media stream. Inanother embodiment, the supplemental content is displayed via adifferent display device (e.g., the media stream is being displayed viaa television or mobile phone, and the supplemental content is displayedon the viewer's laptop computer or is sent to the viewer's e-mailaccount).

In another embodiment, where the supplemental content is displayed as amenu or list of articles appearing in the media stream for whichsupplemental content is available, the mode of display may allow theviewer to “surf” through the mapped supplemental content without havingto view the associated media stream. As illustrated in FIG. 28, whichillustrates the system of FIG. 26 after the viewer has indicatedinterest in a single article for which supplemental content is mapped,the viewer may select any one or more of the articles displayed in themenu for further examination. For example, in this case, the viewer haschosen to request more information about the sweater displayed in FIG.27.

In step 814, the method 800 receives a subsequent signal from theviewer. In one embodiment, the subsequent signal may be received in anyof the ways in which the first signal was received. The method 800 thenproceeds to step 816 and determines whether the subsequent signalindicates that the viewer wishes to return to the media stream (i.e.,exit the currently displayed supplemental content without any furtheraction). If the method 800 concludes in step 816 that the subsequentsignal indicates that the viewer wishes to return to the media stream,the method 800 returns to step 806 and resumes displaying the mediastream (e.g., unobstructed by the supplemental content). Thus, themethod 800 ceases to display the supplemental content until and unlessanother signal to do so is received.

Alternatively, if the method 800 concludes in step 816 that thesubsequent signal does not indicate that the viewer wishes to return tothe media stream, the method 800 proceeds to step 818 and determineswhether the subsequent signal indicates that the viewer wishes topurchase one or more articles represented by the supplemental content.Thus, the method 800 can provide an immediate purchase opportunity ondemand. In one embodiment, the service that hosts the method 800 (e.g.,a stream annotation provider, an access provider or an online accessprovider) receives some revenue each time a viewer sends a signal thatindicates interest in one or more items depicted in the displayedsupplemental content. In a further embodiment still, the access providershares a portion of the revenue with the creator of the media stream(e.g., a content creator) or his/her collaborators, as described above.

If the method 800 concludes in step 818 that the subsequent signal doesnot indicate that the viewer wishes to purchase one or more itemsdepicted in the displayed supplemental content, the method 800 returnsto step 812 and continues to display the supplemental content.Alternatively, if the method 800 concludes in step 818 that thesubsequent signal does indicate that the viewer wishes to purchase oneor more items depicted in the displayed supplemental content, the method800 proceeds to step 820 and consummates the desired transaction. In oneembodiment, the service that hosts the method 800 receives some revenueeach time a viewer purchases one or more items as a result of beingpresented with the supplemental content. In a further embodiment still,the access provider shares a portion of the revenue with the creator ofthe media stream.

Once the transaction is completed, the method 800 returns to step 814and awaits a signal from the viewer indicating how to proceed (e.g.,return to media stream or consummate another transaction). In furtherembodiments, additional options or actions may be associated with thesubsequent signal. For example, the subsequent signal may trigger a pushof additional supplemental content (optionally driven by demographicsand/or learned user preferences, such as reviews by other users withsimilar preferences, consumer reports, etc.). In one embodiment, theadditional supplemental content provides an opportunity for the viewerto view his or her purchase history or to edit preferences relating tothe method 800.

By mapping supplemental content to the segment identifiers anddisplaying the supplemental content only when prompted by the viewer,product placement can be achieved in a manner that does not intrudesubstantially on the viewing experience. That is, the product placementsare substantially invisible until the viewer takes some sort ofaffirmative action. Thus, artistic concerns and viewer expectations canbe accommodated while providing ample opportunities for the presentationof commercial information. In addition, the user interface is much moreuser friendly than interfaces associated with conventional techniques.For example, because supplemental content is mapped to segmentidentifiers, and because the viewer may access the supplemental contentsimply by sending a signal at the appropriate time, there is no need toattempt to click on moving objects or react to “fast cut” editingtechniques.

In one embodiment, the mapping, as well as the identifiers of anysegments in which a viewer has indicated interest, may be cached locallyat a user display device. This enables, for example, the use of portableand/or small form factor display devices, which may be read only or may,at times, lack network access. In this embodiment, the mapping is storedin cache until network connectivity is established. This might occur,for example, by “getting back in range” of a network (in which case, anyportion of a segment identifier that was not previously transmitted istransmitted), or by accessing another device that is connected to thenetwork (e.g., via hot-sync, where the mapping is transferred from theuser device into the sync device, and then sent, for example, to astream annotation service provider).

In one embodiment, segment identifier information is encrypted by theuser device before being sent to the stream annotation service provider(e.g., so that third parties cannot intercept the transmission). Thus,even though the segment identifier may not be able to be encrypted, theactual transmission of the segment identifier can be encrypted. In thisembodiment, the stream annotation provider, access provider, or otherparty providing annotation services is configured to decrypt thesetransmissions and provide the requested supplemental content.

In further embodiments, “hot-spotting” functionality is enabled wherebythe viewer is enabled to select (e.g., by clicking on) a particularimage appearing on the display device, so as to access supplementalcontent about the image (e.g., clicking on a sweater worn by anactress). Current approaches for implementing this type of functionalityare computationally burdensome, time intensive, and costly. In suchsituations, it may be very valuable to be able to efficiently identifythe “boundaries” of the image of a given article within a give segmentof a media stream.

Data collected in accordance with the present invention, particularlywith respect to the classification or description of segments of mediastreams, will enable the prediction not just of articles in which aviewer may be interested and how to best “present” these articles, butalso of subject matter (i.e., types or genres of television, movies andthe like) in which the viewer may be interested. This is particularlysignificant when multiple different platforms are utilized to reviewmedia streams and when media streams are available on-demand, becausecollected data may be utilized to provide (or to highlight to thespecific viewer) media stream subject matter that is likely to be ofinterest. For example, if a given viewer has signaled an interest in anadvertisement relating to tickets for a local baseball team's games,baseball information may be brought to the attention of the viewer, forexample by highlighting upcoming baseball games in program guides, byrecording baseball games as part of a “we thought you might like this”functionality, and the like.

In one embodiment, various “permutations” of a given media stream are“mixed” together either before the media stream begins or as the mediastream is shown to a viewer, based on generated or collected data. Forexample, if the viewer tends to respond to advertising relating toarticles that are categorized as “romance-related,” then the “romantic”version of the media stream may be displayed to the viewer. As anotherexample, the video component of the media stream may be unchanged, butthe audio component including music that is played over particularscenes (or even over the credits) might be selected based on observedviewer preferences (e.g., old school rap, the latest rap hit, hip hop,etc.). Similar data may be used to target product placement within apreviously filmed media stream utilizing computer graphics to replacegeneric article information in the original media stream with specificarticle information. For example, the collected data might indicate thatfor a particular viewer, displaying an actor wearing a Brand X watch(rather than, say, a Brand Y watch) will be more likely to induce theviewer to respond to the supplemental content.

Put another way, the functionality described above may result in thecreation of multiple, potentially overlapping media stream segmentsbeing created under a single media stream heading, such that the optimalmedia stream (e.g., a specific version of a “base” media stream) for aparticular viewer is determined by reference to the viewer's previousresponses to supplemental content. In one embodiment, viewer datadictates the optimal set of sub-streams that are mixed together beforethe media stream is displayed. In one embodiment, the mixing ofsub-streams occurs on the fly, potentially triggered by viewerinteraction (or lack of interaction) with supplemental content featuredearlier in the same media stream.

In one embodiment, a database may include information identifying “sets”of segments that are grouped together as a specific scene. Then, whenthe viewer sends a signal indicating interest in supplemental contentduring a particular scene, one of the options that is displayed to theviewer is “buy this scene for later viewing.” The price charged could bespecific to the viewer. The specific scene being shown would be detectedand sent to the viewer's online account, to his/her digital videorecorder, etc.

Content creators may not want to enable viewers to interrupt thecontinuity of their creative vision (i.e., as embodied in their mediastreams). Similarly, content creators may worry that a viewer jumpingfrom, say, a television show to an e-commerce website might cause a lossof viewership. The mapping associated with a particular media stream mayinclude a code signifying that, with respect to this particular mediastream, mapped supplemental content can be accessed only in certaincontexts. This could be enforced by sending signals to the viewerdisplay device to “lock up” during display of a media stream (e.g.,allow bookmarking of supplemental content only), by sending supplementalcontent only to the viewer's online account or by not allowing thesupplemental content to be provided until after a certain date (e.g., toensure that the critical “word of mouth” audiences in the early weeks ofa release aren't distracted).

In one embodiment, an intermediary device that is able to read segmentidentifiers embedded in a media stream that flows therethrough is placedbetween the media access link (i.e., the access provider) and the viewerdisplay device. When the user signals an interest in receivingsupplemental content, the segment identifier is sent by the intermediarydevice back to the access provider. The intermediary device can alsosend specific information regarding the viewer (which may be obtainedfrom input the viewer provides upon installing the intermediary device).The access provider then identifies and feeds the relevant informationback to the intermediary device and/or to the viewer's online portal.

In one embodiment, the intermediary device sends a signal (via Wi-Fi,bluetooth, etc.) containing the segment identifier. The signal isreceived by a remote control device (e.g., a TV remote, an enabledcellular phone or other device). The remote control device then sendsthe signal (via Wi-Fi, wireless, etc.) directly to the access provider,triggering the processes described above.

In one embodiment, the intermediary device is operated using any one ormore of multiple remote control devices, and the intermediary devicealso sends to a viewer ID code that matches the remote control device tothe access provider, so that specific viewer signaling interest isidentified (e.g., a mother, father, and child may each have their ownremote control device).

In one embodiment, a single remote control device is used for a numberof viewers, but the remote control device is set (e.g., via a touchscreen menu, a button, etc.) to identify the specific viewer using theremote control device at that moment.

In one embodiment, each household or other location has a singleintermediary device deployed at the location to which a media stream isdelivered (a “master” intermediary device), and subsidiary receivers areplaced with each display device in the household or other location toreceive the viewer's signals and to transmit the signals back to themaster intermediary device for transmission back to the access provider(either through the same delivery system, such as Internet ProtocolTelevision (IPTV) or through another delivery system such as viacellular phone signal).

In one embodiment, the access provider would not rely solely on “which”remote control device associated with a given intermediary deviceprovided the a signal, because viewers may choose to use anotherviewer's remote control device for various reasons. Thus, althoughremote control device identification would be the most dominant factorfor determining viewer identity, this information could be combined witha statistical model and/or with viewer ability to override the defaultidentity.

In one embodiment, at a viewer's request, all credit card approvals andsimilarly “sensitive” inputs are preceded by a password input to preventthe consummation of unauthorized transactions.

In one embodiment, speech recognition technology (e.g., using voicerecognition/speaker identification to obtain a voice “fingerprint”) isused in combination with the viewer's voice to activate the intermediarydevice. Thus, the remote control device would initially be programmed todifferentiate among the various viewers who use the remote controldevice.

In one embodiment, the access provider may perform analysis in order todetermine which viewers appear to be viewing media streams together(e.g., multiple members of a household watching television together) sothat supplemental content (which will be customized) is targeted for the“mix” of likely viewers.

In one embodiment, instead of the individual viewer, the probable groupof viewers is determined, which has its own targeted information. Forexample, if a group of viewers is predicted to be a hip-hop music lovingson and a classical music loving father, then the supplemental contentassociated with a media stream being viewed by the group of users mightbe targeted to a more neutral demographic (rather than to either a hiphop demographic or a classical demographic).

In one embodiment, voice recognition technology may be used to improveone-click capabilities. Given a limited set of viewer voices associatedwith a given remote control device, one voice can be very reliablydifferentiated from other voices. This could be used to facilitatepurchases—instead of having to type in a password or provide some otherverification, each viewer's profile could be set to authorize onlycertain transactions, such as access to viewing certain types of mediastreams.

In one embodiment, the fact that an authorizing party authorized viewingof a particular media stream by other parties (e.g., a parentauthorizing a media stream for viewing by his/her children) is recordedin some form and provided as verification to a third party that theauthorizing party was viewing the program with the other parties. Suchdetermination may be sold as a service to schools, for example. Themedia stream could prompt the authorizing party, at different times, fora vocal response in order to prevent the authorizing party from simplyinitiating the media stream and then leaving.

Some viewers have a strong aversion to advertising (or to certainadvertisers), but might appreciate many of the other features that thepresent invention can enable. To this end, a viewer may be enabled toblock particular types of supplemental content that he/she does not wantto see (e.g., an animal rights activist may wish to block advertisementsfrom fast food chains).

In one embodiment, prior to sending a mapped item of supplementalcontent back to a viewer in response to a signal, the present inventionsends the supplemental content through a filter (perhaps created by theviewer or by the access provider) and removes any undesired information.For example, the filter may remove advertisements for particulararticles, advertisements featuring particular types of music orsupplemental content of certain genres (e.g., violent or adult-themedcontent). In one embodiment, this functionality could be used as amethod of parental control.

In one embodiment involving video on demand services, one could use adatabase of filtered content to actually remove offending segments “onthe fly” from media streams (e.g., using or segment-specific informationand/or input from the content creator). This would enable differentversions of the media stream (e.g., PG, PG-13, etc.) to be displayed todifferent viewers.

FIG. 9 is a flow diagram illustrating a second embodiment of a method900 for providing annotated media streams to viewers, according to thepresent invention. The method 900 may be implemented, for example, by auser device associated with the display of media streams. In particular,the method 900, unlike the method 800, allows for “on the fly”annotation (e.g., without the assistance of a pre-determined mapping).

The method 900 is initialized at step 902 and proceeds to step 904,where the method 900 displays the media stream via a display device. Themethod 900 then proceeds to step 906 and receives a first signal from aviewer of the media stream. The first signal indicates that the viewerwishes to view supplemental content relating to at least one articleappearing in the media stream. In one embodiment, the first signal isconveyed via the push of a button or via a spoken command. In oneembodiment, the method 900, like the method 800, supports multimodalviewer interaction. In one embodiment, the first signal indicates thatthe viewer wishes to view the supplemental content immediately. Inanother embodiment, the first signal indicates that the viewer wishes toview the supplemental content at a later time.

In step 908, the method 900 identifies, in substantially real time, oneor more articles appearing in the media stream at or around the point intime at which the first signal was received (i.e., what portion of themedia stream was being displayed when the viewer sent the first signal).In one embodiment, the method 900 identifies these articles by applyingone or more optical character recognition (OCR) techniques to the imagesassociated with the indicated portion of the media stream (e.g., videoOCR, text recognition, object recognition, logo recognition, etc.). Inthis manner, text, logos, objects and individuals appearing in the mediastream (and in which the viewer may be interested) can be identified.

The method 900 then proceeds to step 910 and retrieves supplementalcontent relating to one or more of the identified articles. In oneembodiment, this supplemental content may be retrieved from a remoteserver or database such as a shopping or information portal (e.g., anInternet search engine or e-commerce World Wide Web site), for examplevia a connection through the Internet or a cable network. For example,the method 900 might perform a search on recognized text using anInternet search engine. In another embodiment, the remote server ordatabase stores or has access to a mapping that maps supplementalcontent to segment identifiers identifying segments of the media stream(as described above). In this case, the remote server matches one ormore segments identifiers identifying segments that correspond to viewersignals to the appropriate supplemental content.

In step 912, the method 900 displays, at the appropriate time (e.g.,based on whether the user wishes to view the supplemental contentimmediately or at a later time), at least a portion of the retrievedsupplemental content. Thus, for instance, if the first signal wasreceived at a time when one or more segments of the media signaldepicting credits (e.g., for a movie) were displayed, the method 900might request supplemental content based on recognition of namesappearing in the credits, and accordingly convert one or more of thecredits into hyperlinks linked to additional information (e.g., a moviedatabase entry) about the named individual(s).

For example, FIG. 10 is a schematic diagram illustrating an exemplarydisplay 1000 for presenting retrieved supplemental content aboutarticles appearing in a portion of a media stream. The display 1000 maybe conveyed via a display device as described above. The display 1000 isarranged in a menu format and includes one or more images 1002 ₁-1002_(n) (hereinafter collectively referred to as “images 1002”) depictingarticles appearing in an associated portion of a media stream (e.g.,clothing, beverages, compact discs). In one embodiment, the images 1002include hyperlinks such that a viewer may retrieve supplemental contentabout the depicted article (e.g., content, price, purchasingopportunities) by selecting the article's associated image 1002. Theselection of an image 1002 may be made by sending a signal to a devicethat retrieves the supplemental content.

In one embodiment, the display 1000 is prioritized according to somecriteria (e.g., popularity with similar viewers, fees paid by sellers,etc.). For example, an image 1002 depicting an article that isstatistically determined to be the article in which the viewer is mostlikely to be interested (e.g., “78% of viewers who sent a signal duringframes X-Z clicked though to Y brand sneakers”) may be displayed mostprominently. Alternatively, article images may be displayed according tohow much the sellers are willing to pay to advertise the articles (e.g.,the more the seller pays for the advertising, the more prominently thearticle image is displayed). In one embodiment, the n highest priorityarticles are displayed as images, while the remainder of the articlesdepicted in the menu are displayed as text hyperlinks or grouped intoone or more sub-categories.

In one embodiment, articles depicted in a menu display are prioritizedby providing each article with a score, such that the scores of multiplearticles may be compared to determine the relative priorities of thearticles. In one embodiment, the score is a weighted combination offactors including at least one of: the article's level of popularitywith viewers, the likelihood that a given viewer is interested in thearticle and the fees paid by the article's seller for eachclick-through. In one embodiment, the score, S, of an article for thepurposes of determining priority in a menu of articles is calculatedaccording to:S=Cw ₁ +Vw ₂ +Pw ₃where C is the cost per click-through paid by a seller when the articleis selected from a menu by a viewer, V is a statistical measure of agiven viewer's likely interest in the article (based, for example, onthe viewer's other observed interests) and P is a measure of thearticle's popularity with viewers in general. w1, w2 and w3 are weighingfactors that may be selected, for example, by a provider of annotatedmedia streams and associated services (e.g., a stream annotation serviceprovider or an access provider).

Referring back to FIG. 9, in step 914, the method 900 receives asubsequent signal from the viewer. In one embodiment, the subsequentsignal may be received in any of the ways in which the first signal wasreceived. The method 900 then proceeds to step 916 and determineswhether the subsequent signal indicates that the viewer wishes to returnto the media stream (i.e., exit the currently displayed supplementalcontent without any further action). If the method 900 concludes in step916 that the subsequent signal indicates that the viewer wishes toreturn to the media stream, the method 900 returns to step 904 andresumes displaying the media stream (e.g., unobstructed by thesupplemental content). Thus, the method 900 ceases to display theretrieved information supplemental content until and unless anothersignal to do so is received.

Alternatively, if the method 900 concludes in step 916 that thesubsequent signal does not indicate that the viewer wishes to return tothe media stream, the method 900 proceeds to step 918 and determineswhether the subsequent signal indicates that the viewer wishes topurchase one or more articles depicted in the supplemental content.Thus, the method 900 can provide an immediate purchase opportunity ondemand.

If the method 900 concludes in step 918 that the subsequent signal doesnot indicate that the viewer wishes to purchase one or more articlesdepicted in the supplemental content, the method 900 returns to step 912and continues to display the supplemental content. Alternatively, if themethod 900 concludes in step 918 that the subsequent signal doesindicate that the viewer wishes to purchase one or more articlesdepicted in the supplemental content, the method 900 proceeds to step920 and consummates the desired transaction (e.g., via an Internete-commerce site). Once the transaction is completed, the method 900returns to step 914 and awaits a signal from the viewer indicating howto proceed (e.g., return to media stream or consummate anothertransaction).

In further embodiments, additional options or actions may be associatedwith the subsequent signal. For example, the subsequent signal maytrigger a push of additional supplemental content (optionally driven bydemographics and/or learned user preferences, such as reviews by otherusers with similar preferences, consumer reports, etc.). In oneembodiment, the additional supplemental content provides an opportunityfor the viewer to view his or her purchase history or to editpreferences relating to the method 900.

FIG. 11 is a flow diagram illustrating a third embodiment of a method1100 for providing annotated media streams to viewers, according to thepresent invention. Specifically, the method 100, unlike the methods 800and 900 described above, is configured to support processing of spokenviewer signals. Like the methods 800 and 900, the method 1100 may beimplemented by a user device (e.g., a television, a DVD player, a DVR, aset-top box, a portable media player, a personal digital assistant, apersonal computer, a mobile phone, an in-flight or in-vehicleentertainment system, a video game console or the like) that has accessto annotated media streams.

The method 1100 is initialized at step 1102 and proceeds to step 1104,where the method 1100 receives a mapping of one or more keywords to oneor more items of supplemental content. In one embodiment, thesupplemental content is associated with commercial information relatingto one or more articles appearing in the media stream, as describedabove. In one embodiment, the mapping is received from a remote server,for example, via the Internet, via a cable network (e.g., where theremote server may be the cable head-end, or the head-end may mediatecommunications with the remote server) or via satellite. Particularembodiments may receive the mapping as an OpenCable application platform(OCAP) or digital video broadcasting (DVB) application, or as a privatesection within a Motion Picture Experts Group (MPEG) transport stream.In a further embodiment, the mapping is received “on the fly” from a“director” who views the media stream in substantially real time andmaps supplemental content from a pre-set list to the keywords. In oneembodiment, the mapping further includes the URL of one or more WorldWide Web sites relating to one or more articles appearing in the mediastream (e.g., to reduce subsequent lookup times).

In step 1106, the method 1100 displays the media stream via a displaydevice. In one embodiment, displaying the media stream includespresenting an indicator when the segment identifier identifying thecurrently displayed portion of the media stream (e.g., the currentlydisplayed segment) is associated with at least one item of supplementalcontent. Thus, a viewer of the media stream would automatically knowwhen additional information about one or more articles being displayedis available (e.g., as opposed to having to guess or query). In oneembodiment, the indicator is presented when the currently displayedportion of the media stream is associated with one or more items ofsupplemental content relating to information that is expected to be ofinterest to the viewer (e.g., based on demographic information, statedviewer preferences or previous viewer activity). In one embodiment, theindicator is a visual indicator, such as an icon displayed in the mediastream or a (perhaps flashing) light on the media stream player (e.g., aset-top box, DVR or the like), on the display device or on a remotecontrol. In further embodiments, the indicator may be an audibleindicator or other sensory indicator (e.g., a vibrating remote control).In a further embodiment still, a “split screen” display may be employedthat displays the media stream in a first portion of the display anddata relating to currently accessible supplemental content in a secondportion of the display.

The method 1100 then proceeds to step 1108 and receives a first speechsignal from a viewer of the media stream. The first speech signalindicates that the viewer wishes to view supplemental content relatingto one or more articles appearing in the media stream. In oneembodiment, the first speech signal indicates that the viewer wishes toview the supplemental content immediately (e.g., “show car now”). Inanother embodiment, the first signal indicates that the viewer wishes toview the supplemental content at a later time, such as at the conclusionof the media stream (e.g., “save car for later”). In furtherembodiments, the first speech signal indicates that the user wishes tosave the present position in the media stream to view the supplementalcontent at a later time, without specifically identifying the item ofinterest (e.g., “mark for later”).

In step 1110, the method 1100 processes the first speech signal torecognize a viewer command embodied therein. In one embodiment, thefirst speech signal is processed in accordance with one or moreautomatic speech recognition and/or natural language understandingtechniques. For example, in one embodiment, the first speech signal isprocessed in accordance with an automatic speech recognition techniquethat operates in conjunction with a defined vocabulary. In refinementsof this embodiment, the defined vocabulary comprises the keywordscontained in the mapping received at step 1104. In particularembodiments, the defined vocabulary is dynamically modifiable as themedia stream is displayed, such that the vocabulary comprises the subsetof the keywords that are presently “valid”. In another embodiment, thefirst speech signal is processed in accordance with an automatic speechrecognition technique that cancels ambient noise, such as thatassociated with the media stream itself (e.g., such as the methoddescribed in U.S. patent application Ser. No. 11/541,282, filed Sep. 28,2006, which is herein incorporated by reference in its entirety). Inanother embodiment, the first speech signal is processed in accordancewith a “light-weight” natural language processing technique such astopic spotting.

In one embodiment, the method 1100 supports multimodal viewerinteraction, meaning that one or more additional input modalities (inaddition to speech) may be employed for receiving viewer signals.Multimodal interaction may be helpful, for example, in correcting errorsin processing speech signals. For example, errors in processing speechsignals may be corrected in accordance with the methods described inU.S. Pat. No. 6,742,021, which is herein incorporated by reference inits entirety.

In step 1112, the method 1100 matches the viewer command to one or morekeywords used in the mapping. In this way, the method 1100 can identifywhich item(s) of supplemental content corresponds to the keyword(s)uttered by the viewer. In one embodiment, if the viewer command cannotbe matched to a known keyword, the method 1100 uses the viewer commandto query a database such as a shopping or information portal (e.g., anInternet search engine or e-commerce Word Wide Web site). In this way,the viewer command can still be potentially matched to relevantsupplemental content. In a further embodiment, the viewer command isparsed by a natural language understanding technique that chooses anappropriate portal to query. In certain embodiments, the step ofsearching a database is performed in accordance with the methodsdescribed in U.S. patent application Ser. No. 10/912,517, filed Aug. 5,2004; and Ser. No. 11/063,357, filed Feb. 23, 2005, both of which areherein incorporated by reference in their entireties.

The method 1100 then proceeds to step 1114 and displays, at theappropriate time (e.g., based on whether the user wishes to view thesupplemental content immediately or at a later time), the supplementalcontent that is mapped to the corresponding keyword. For example, if theuttered keyword is “car” or “automobile”, the displayed information (asembodied in the supplemental content) might include the maker and/ordealers associated with a car appearing in the media stream. In oneembodiment, the display of the media stream is paused while thesupplemental content is displayed. In one embodiment, the supplementalcontent is displayed via the same display device as the media stream. Inanother embodiment, the information is displayed via a different displaydevice (e.g., the media stream is being displayed via a television ormobile phone, and the supplemental content is displayed on the viewer'slaptop computer or is sent to the viewer's e-mail account).

In a further embodiment (e.g., where the keyword is ambiguous orpotentially corresponds to any one of multiple articles), theinformation is displayed in step 1114 in the form of a menu of two ormore selectable articles. Thus, for instance, if the keyword “car” isuttered at a time when four different cars appear in the media stream,the supplemental content mapped to the keyword might include images ofall four cars, each image being further associated with a respective,unambiguous keyword (e.g., “car A”, “car B”, “car C” and “car D”). Themetadata mapped to each unambiguous keyword could include links to WorldWide Web sites for the respective cars' manufacturers and/or dealers. Inthis manner, the method 1100 supports disambiguation.

In another embodiment, the information is displayed as a menu or list ofarticles appearing in the media stream for which information isavailable (and may be associated, for example, with a keyword such as“menu”). This would allow a viewer to “surf” through the mappedsupplemental content without having to view the associated media stream.

In step 1116, the method 1100 receives a subsequent signal from theviewer. In one embodiment, the subsequent signal may be received in theform of a speech signal. In another embodiment, the subsequent signalmay be received in a different manner, such as via a keystroke or thepush of a button on a remote control. The method 1100 then proceeds tostep 1118 and determines whether the subsequent signal indicates thatthe viewer wishes to return to the media stream (i.e., exit thecurrently displayed supplemental content without any further action). Ifthe method 1100 concludes in step 1118 that the subsequent signalindicates that the viewer wishes to return to the media stream, themethod 1100 returns to step 1106 and resumes displaying the media stream(e.g., unobstructed by the supplemental content). Thus, the method 1100ceases to display the supplemental content until and unless anothersignal to do so is received.

Alternatively, if the method 1100 concludes in step 1118 that thesubsequent signal does not indicate that the viewer wishes to return tothe media stream, the method 1100 proceeds to step 1120 and determineswhether the subsequent signal indicates that the viewer wishes topurchase one or more articles depicted in the displayed supplementalcontent. Thus, the method 1100 can provide an immediate purchaseopportunity on demand.

If the method 1100 concludes in step 1120 that the subsequent signaldoes not indicate that the viewer wishes to purchase one or morearticles depicted in the displayed supplemental content, the method 1100returns to step 1114 and continues to display the supplemental content.Alternatively, if the method 1100 concludes in step 1120 that thesubsequent signal does indicate that the viewer wishes to purchase oneor more articles depicted in the displayed supplemental content, themethod 1100 proceeds to step 1122 and consummates the desiredtransaction. Once the transaction is completed, the method 1100 returnsto step 1116 and awaits a signal from the viewer indicating how toproceed (e.g., return to media stream or consummate anothertransaction). In further embodiments, additional options or actions maybe associated with the subsequent signal. For example, the subsequentsignal may trigger a push of additional supplemental content (optionallydriven by demographics and/or learned user preferences, such as reviewsby other users with similar preferences, consumer reports, etc.). In oneembodiment, the additional supplemental content provides an opportunityfor the viewer to view his or her purchase history or to editpreferences relating to the method 1100.

FIG. 12 is a flow diagram illustrating a fourth embodiment of a method1200 for providing annotated media streams to viewers, according to thepresent invention. The method 1200 may be implemented, for example, by auser device associated with the display of media streams. In particular,the method 1200, unlike the method 1100, allows for speech-supported “onthe fly” annotation (e.g., without the assistance of a pre-determinedmapping).

The method 1200 is initialized at step 1202 and proceeds to step 1204,where the method 1200 displays the media stream via a display device.The method 1200 then proceeds to step 1206 and receives a first speechsignal from a viewer of the media stream. The first speech signalindicates that the viewer wishes to view supplemental content about atleast one article appearing in the media stream. In one embodiment, thefirst speech signal indicates that the viewer wishes to view thesupplemental content immediately. In another embodiment, the firstspeech signal indicates that the viewer wishes to view the supplementalcontent at a later time.

In step 1208, the method 1200 processes, in substantially real time, thefirst speech signal in order to recognize a viewer command embodiedtherein. In one embodiment, the first speech signal is processed inaccordance with one or more automatic speech recognition and/or naturallanguage understanding techniques. For example, in one embodiment, thefirst speech signal is processed in accordance with an automatic speechrecognition technique that operates in conjunction with a definedvocabulary. In another embodiment, the first speech signal is processedin accordance with an automatic speech recognition technique thatcancels ambient noise, such as that associated with the media streamitself (e.g., such as the method described in U.S. patent applicationSer. No. 11/541,282, filed Sep. 28, 2006, which is herein incorporatedby reference in its entirety). In another embodiment, the first speechsignal is processed in accordance with a “light-weight” natural languageprocessing technique such as topic spotting.

In one embodiment, the method 1200, like the method 1100, supportsmultimodal viewer interaction, meaning that one or more additional inputmodalities (in addition to speech) may be employed for receiving viewersignals. Multimodal interaction may be helpful, for example, incorrecting errors in processing speech signals. For example, errors inprocessing speech signals may be corrected in accordance with themethods described in U.S. Pat. No. 6,742,021, which is hereinincorporated by reference in its entirety.

The method 1200 then proceeds to step 1210 and queries a remote serverin accordance with the recognized viewer command, for example via aconnection through the Internet or a cable network. For example, themethod 1200 might perform a search on recognized words in the viewercommand using a database such as a shopping or information portal (e.g.,an Internet search engine or e-commerce World Wide Web site). In oneembodiment, querying in accordance with step 1210 includes selecting anappropriate query portal (e.g., search engine or e-commerce World WideWeb site) based on the recognized words. In another embodiment, theremote database stores or has access to a mapping that maps supplementalcontent to keywords (as described above). In this case, the remoteserver matches one or more words in the recognized viewer command tosupplemental content. In certain embodiments, the step of searching isperformed in accordance with the methods described in U.S. patentapplication Ser. No. 10/912,517, filed Aug. 5, 2004; and Ser. No.11/063,357, filed Feb. 23, 2005, both of which are herein incorporatedby reference in their entireties.

In step 1212, the method 1200 displays, at the appropriate time (e.g.,based on whether the user wishes to view the supplemental contentimmediately or at a later time), at least a portion of the queryresults. Thus, for instance, if the first speech signal was received ata time when one or more segments of the media signal depicting credits(e.g., for a movie) were displayed, and an actor's name was recognizedin the speech recognition processing step (i.e., step 1208), the method1200 might display supplemental content (e.g., a movie database entry)about the actor.

In step 1214, the method 1200 receives a subsequent signal from theviewer. In one embodiment, the subsequent signal is a speech signal. Inanother embodiment, the subsequent signal is a non-speech signal, suchas a keystroke or button press. The method 1200 then proceeds to step1216 and determines whether the subsequent signal indicates that theviewer wishes to return to the media stream (i.e., exit the currentlydisplayed supplemental content without any further action). If themethod 1200 concludes in step 1216 that the subsequent signal indicatesthat the viewer wishes to return to the media stream, the method 1200returns to step 1204 and resumes displaying the media stream (e.g.,unobstructed by the supplemental content). Thus, the method 1200 ceasesto display the retrieved supplemental content until and unless anothersignal to do so is received.

Alternatively, if the method 1200 concludes in step 1216 that thesubsequent signal does not indicate that the viewer wishes to return tothe media stream, the method 1200 proceeds to step 1218 and determineswhether the subsequent signal indicates that the viewer wishes topurchase one or more articles depicted in the displayed supplementalcontent. Thus, the method 1200 can provide an immediate purchaseopportunity on demand.

If the method 1200 concludes in step 1218 that the subsequent signaldoes not indicate that the viewer wishes to purchase one or morearticles depicted in the displayed supplemental content, the method 1200returns to step 1212 and continues to display the query results.Alternatively, if the method 1200 concludes in step 1218 that thesubsequent signal does indicate that the viewer wishes to purchase oneor more articles depicted in the displayed supplemental content, themethod 1200 proceeds to step 1220 and consummates the desiredtransaction (e.g., via a retrieved Internet e-commerce site). Once thetransaction is completed, the method 1200 returns to step 1214 andawaits a signal from the viewer indicating how to proceed (e.g., returnto media stream or consummate another transaction). In furtherembodiments, additional options or actions may be associated with thesubsequent signal. For example, the subsequent signal may trigger a pushof additional supplemental content (optionally driven by demographicsand/or learned user preferences, such as reviews by other users withsimilar preferences, consumer reports, etc.). In one embodiment, theadditional information provides an opportunity for the viewer to viewhis or her purchase history or to edit preferences relating to themethod 1200.

Those skilled in the art will appreciate that the methods of the presentinvention are not limited to the field of advertising. For instance,media stream segment identifiers may be pulled, in accordance with thepresent invention, to track certain parameters occurring at a precisemoment in time or under precise conditions or viewer criteria. Forexample, the methods of the present invention may be adapted to identifya “statistically significant” sample (e.g., national or demographic- orsubset-specific) to determine the viewing habits of a given population.Alternatively, the methods of the present invention may be adapted todetermine when viewers “switch away” from a television show (e.g., bypulling segment identifiers every x seconds). In one embodiment, thesegment identifiers occurring over a given period of time are stored ina user device (e.g., a set top box). In another embodiment, the userdevice is programmed to poll segment identifiers at given times or at agiven rate. The user device may then transmit these segment identifiersto a third party (e.g., a stream annotation service provider or accessprovider) for further analysis.

In further embodiments, part of the supplemental content that is mappedto a segment identifier (and stored, for example, by a stream annotationservice provider or access provider) comprises software that triggers amini application or instruction set. For example, a trigger sent inresponse to a segment identifier might instruct a viewer set top box toactivate a voting mini application in which the viewer is asked to votefor a favorite contestant on a game show. This software could beactivated automatically (e.g., as frame x is displayed) or upon viewersignal (e.g., when the viewer sends a signal during frames x-z).

As discussed above, embodiments of the present invention may be extendedto log viewer activity (e.g., for data mining purposes). For example,viewer requests for information about articles depicted in the mediastreams may be tracked, as may any viewer purchases made as a result ofpresenting the requested supplemental content. This data can assist incustomizing the supplemental content that is made available inconnection with given media streams. For example, the supplementalcontent made available could be customized according to a viewer's age,location, prior purchasing habits, viewing habits and the like.Moreover, the data can assist in determining a fee structure for when,where and how to place supplemental content relating to a given productor service.

FIG. 15 is a flow diagram illustrating a fifth embodiment of a method1500 for providing annotated media streams to viewers, according to thepresent invention. The method 1500 may be implemented, for example, inits entirety by the access provider 106 of FIG. 1 (e.g., via a userdevice such as a television, a DVD player, a DVR, a set-top box, aportable media player, a personal digital assistant, a personalcomputer, a mobile phone, an in-flight or in-vehicle entertainmentsystem, a video game console, a dedicated device or the like that hasaccess to annotated media streams). Alternatively, the method 1500 maybe implemented by access provider 106 in conjunction with the streamannotation service provider 112. In particular, the method 1500 isintended for providing annotated media streams in which segmentidentifiers identifying segments of the media streams are mapped tointermediate nodes, the intermediate nodes being in turn mapped to itemsof supplemental content (e.g., as described above).

The method 1500 is initialized at step 1502 and proceeds to step 1504,where the method 1500 receives a media stream. In embodiments wheresegment identifiers identifying segments of the media stream are mappedto intermediate nodes, the media stream also contains metadata,describing the segments that individual frames of the media streambelong to. In one embodiment, the metadata is not contained within themedia stream, but is instead received from a remote server, for example,via the Internet, via a cable network (e.g., where the remote server maybe a cable head-end, or the head-end may mediate communications with theremote server) or via satellite. Particular embodiments may receive themapping as an OpenCable application platform (OCAP) or digital videobroadcasting (DVB) application, or as a private section within an MPEGtransport stream.

In step 1506, the method 1500 displays the media stream via a displaydevice. The method 1500 then proceeds to step 1508 and receives a firstsignal from a viewer of the media stream. The first signal indicatesthat the viewer wishes to view any supplemental content that may beassociated with the present position in the media stream (i.e., that ismapped to the corresponding segment identifier). The first signal may beconveyed via the push of a button (e.g., on a remote control, akeyboard, a mouse, a light pen or a laser pointer) or via a spokencommand (e.g., to a microphone, a mobile phone or a voice-over-IPhandset). In one embodiment, the method 1500 supports multimodal viewerinteraction, meaning that two or more input modalities may be employedfor receiving viewer signals. Multimodal interaction may be helpful, forexample, in correcting errors in processing viewer signals. For example,errors in processing viewer signals may be corrected in accordance withthe methods described in U.S. Pat. No. 6,742,021, which is hereinincorporated by reference in its entirety. In one embodiment, the firstsignal indicates that the viewer wishes to view the supplemental contentimmediately. In another embodiment, the first signal indicates that theviewer wishes to view the supplemental content at a later time (e.g., atthe conclusion of the media stream). In further embodiments, the firstsignal indicates that the user wishes to save the present position inthe media stream to view the supplemental content at a later time,without specifically identifying the article of interest (e.g., “markfor later”).

In step 1510, the method 1500 determines, in accordance with themetadata, which segment identifier corresponds to the point in time atwhich the first signal was received (i.e., what portion of the mediastream was being displayed when the viewer sent the first signal). Inone embodiment (e.g., where the segment identifiers comprisehash-generated identifiers), this determination is made by computing thehash of the current frame, and identifying the segment associated withthe computed hash. In embodiments where time offset (either from thebeginning of the media stream or from some internal boundary) is used asthe index for segments, metadata need not be provided, and an internalclock of the display device or associated hardware may be utilized tocalculate the running time of the media stream. In another embodiment, atime-code datum or other internal information already present in thecurrent frame is used to determine the present segment and correspondingsegment identifier.

In step 1511, the method 1500 then determines, in accordance with themapping, what supplemental content is mapped from the current segmentidentifier. Step 1511 may be performed directly by the user's device, ormay be performed by a remote computer upon request by the user's device.First, the intermediate node mapped from the current segment identifieris obtained (e.g., via a first mapping). Then, in one embodiment, itemsof supplemental content mapped from the obtained intermediate node(e.g., via a second mapping) are retrieved for display in step 1512.

The method 1500 next proceeds to step 1512 and displays, at theappropriate time (e.g., based on whether the user wishes to view thesupplemental content immediately or at a later time), the retrievedsupplemental content. In one embodiment (e.g., where there are multipleretrieved items of supplemental content), the supplemental content isdisplayed in the form of a menu of selectable articles. Thus, forinstance, if the first signal was received at a time when one or moreframes of the media stream depicting an actress were displayed, thesupplemental content mapped (indirectly) from those frames might includethe actress's filmography, and information about a sweater that theactress is wearing. Thus, the supplemental content displayed in responseto the first signal might include an image of the actress and an imageof the sweater, where each of the images may be associated with ahyperlink to a World Wide Web page, containing additional informationand an opportunity to purchase goods or services. In one embodiment, thedisplay of the media stream is paused while the retrieved supplementalcontent is displayed. In one embodiment, the retrieved supplementalcontent is displayed via the same display device as the media stream. Inanother embodiment, the retrieved supplemental content is displayed viaa different display device (e.g., the media stream is being displayedvia a television or mobile phone, and the retrieved supplemental contentis displayed on the viewer's laptop computer, or is sent to the viewer'se-mail account).

In step 1514, the method 1500 receives a subsequent signal from theviewer. In one embodiment, the subsequent signal may be received in anyof the ways in which the first signal was received. The method 1500 thenproceeds to step 1516 and determines whether the subsequent signalindicates that the viewer wishes to return to the media stream (i.e.,exit the currently displayed supplemental content without any furtheraction). If the method 1500 concludes in step 1516 that the subsequentsignal indicates that the viewer wishes to return to the media stream,the method 1500 returns to step 1506 and resumes displaying the mediastream (e.g., unobstructed by the supplemental content). Thus, themethod 1500 ceases to display the supplemental content until and unlessanother signal to do so is received.

Alternatively, if the method 1500 concludes in step 1516 that thesubsequent signal does not indicate that the viewer wishes to return tothe media stream, the method 1500 proceeds to step 1518 and determineswhether the subsequent signal indicates that the viewer wishes topurchase one or more articles depicted in the displayed supplementalcontent. Thus, the method 1500 can provide an immediate purchaseopportunity on demand.

If the method 1500 concludes in step 1518 that the subsequent signaldoes not indicate that the viewer wishes to purchase one or morearticles depicted in the displayed supplemental content, the method 1500returns to step 1512 and continues to display the supplemental content.Alternatively, if the method 1500 concludes in step 1518 that thesubsequent signal does indicate that the viewer wishes to purchase oneor more articles depicted in the displayed supplemental content, themethod 1500 proceeds to step 1520 and consummates the desiredtransaction. Once the transaction is completed, the method 1500 returnsto step 1514 and awaits a signal from the viewer indicating how toproceed (e.g., return to media stream or consummate anothertransaction). In further embodiments, additional options or actions maybe associated with the subsequent signal. For example, the subsequentsignal may trigger a push of additional supplemental content (optionallydriven by demographics and/or learned user preferences, such as reviewsby other users with similar preferences, consumer reports, etc.). In oneembodiment, the additional supplemental content provides an opportunityfor the viewer to view his or her purchase history or to editpreferences relating to the method 1500.

The method 1500 continues processing, awaiting signals from the user,until the display of the media stream is complete.

As noted earlier, a major disadvantage of prior art approaches is thatthe information (e.g., supplemental content) relating to productplacements in media streams tend to be fixed. There is nostraightforward way to provide differentiated information regardingarticles depicted in a media stream to a viewer of the media streambased on criteria such as demographic, gender, and the like. Further, ifno particular information is associated with a particular video segment,there is no way of calculating what items of information may be mostrelevant and compelling. These significant disadvantages are overcome inthe present approach by calculation over the ontology of intermediatenodes.

For example, consider a situation where a particular segment identifieris mapped via a first mapping to the intermediate node representing theontology instance “Brand X Strawberry Jelly,” and there is no item ofsupplemental content mapped from this intermediate node via a secondmapping. However, there is an item of supplemental content, “GroceryStore A Coupon” mapped to the intermediate node representing theontology concept “Jelly.” If the ontology in this example contains“is-a” relationships, such that the instance “Brand X Strawberry Jelly”is understood to be a member of the class “Jelly,” then the method 1500can calculate that, with respect to the ontology, returning thesupplemental content “Grocery Store A Coupon” is appropriate when amapping to the intermediate node “Brand X Strawberry Jelly” ispresented.

In general therefore, in certain embodiments, when the method 1500determines that a segment identifier is mapped to an intermediate node Nvia a first mapping, and that the intermediate node N has no secondmapping to any articles (or perhaps a second mapping that is undesirableunder constraints, as discussed later), items of supplemental contentcan be searched for by, in effect, “walking” the ontology to locate anitem. By walking “up” the ontology from instances to classes, and fromclasses to super-classes, items of supplemental content may be locatedthat are mapped to a more general concept, as in the earlier example of“Jelly”. There, by proceeding from the specific instance node to thecontaining class, the coupon is located and may then be returned to theuser. Thus, the relationships between instances and classes maintainedin the ontology enable a computation of the item to be returned to theuser.

Inversely from the example above, the method 1500 can also walk “down”the ontology from classes to sub-classes, and from classes to instances,to provide an item of supplemental content mapped from an intermediatenode representing a more specific part of the ontology. For example,assume a segment identifier is mapped via a first mapping to anintermediate node representing an ontology class “Paint,” with a colorattribute set to “Green,” and an instance of this class is present thatrepresents “Brand X Lime Green Paint,” with an associated item ofsupplemental content. By walking the ontology to more specific items,the method can locate and retrieve this item of supplemental content.Further, one can also “walk” to siblings of an instance or class (i.e.,those instances and classes that share a common immediate super-class)in a similar manner, and combine up, down, and sibling walks in variouscombinations. Constraints are preferentially provided to control how farfrom a mapped node to a sibling node the method is permitted to walk.For example, it may be too remote to walk up two super-classes in aparticular ontology, meaning that it would be difficult for a user tocomprehend why the method retrieved a particular item of supplementalcontent. The precise constraints are established according to exerciseof judgment, in reference to a particular ontology used in a particularembodiment.

In ontologies where a “part-of” relationship is provided in addition toan “is-a” relationship, search for items of supplemental content toreturn can also include matching a whole from its parts. For example,consider a segment identifier, S, that is mapped to two intermediatenodes, representing the ontology concepts “Crunchy Peanut Butter,” and“Strawberry Jelly,” respectively. The example ontology provides “is-a”relationships, such that it is known that “Crunchy Peanut Butter” ispart of the class “Peanut Butter,” and “Strawberry Jelly,” is part ofthe class “Jelly.” Further, the example ontology provides a “part-of”relationship such that the classes “Peanut Butter,” and “Jelly,” are thetwo parts of the class “Peanut Butter and Jelly Sandwich.” In searchingfor items of supplemental content, when walking the ontology asdescribed earlier to super-classes, if the class relationship is a“part-of” relationship, the choice to visit the super-class may be madein several ways. First, the super-class may be walked to only if atleast one item from each part subclass is present (e.g., only return anitem linked to “Peanut Butter and Jelly Sandwich” if at least one memberof each of “Jelly” and “Peanut Butter” is mapped from the presentsegment identifier). Alternatively, the super-class may be walked to ifany of the parts are mapped from the present segment identifier.

In the embodiments described so far, it has been implicitly assumed thatthe mapping from an intermediate node to an item of supplemental contentwas applicable in all circumstances. However, in some embodiments thecapacity to present a particular item from a choice between items isdesirable (e.g., to provide item A to teenagers and the alternative itemB to younger children). Such a choice can be accomplished by mapping anintermediate node to many items of supplemental content, and providingconstraints on the mappings. For example, consider an intermediate nodeN that is mapped to three items of supplemental information P1, P2, andP3. Each such mapping has associated constraints, C1, C2, and C3. In afirst approach, the constraints are interpreted as Boolean conditionals,such that the meaning of the constrained mappings is “IF C1 THEN P1,”and so on. The constraints do not need to be mutually exclusive,allowing more than one item of supplemental content to be returned forany particular set of conditions. Further, the constraints do not needto be exhaustive—under a particular set of conditions, no constraint mayevaluate to true, and no item of supplemental content may be returned. A“default” constraint may also be supplied, such that if no other item ofsupplemental content is returned under the conditions, the default itemwill be returned. The constraints can include Boolean operators,numerical and string comparators, and have multiple subterms.

Constraints are contemplated to include variables describing knowledgeregarding the viewer viewing the media stream, including demographics,device type, and the like. For example, constraints may include“viewer's age >30,” “viewer's income >$100,000,” “viewer's zipcode=94025,” “viewer's language=Español,” “viewer's device=DVR orcellular telephone.” Constraints can also include the attributesassociated with a particular instance or class in the ontology.

In further embodiments, constraints can be probabilistic. This allowsfor the situation where information about the viewer is not known, buthas been inferred from viewer behavior by learning approaches. Learningcan be applied to a viewer's viewing habits, click through andpurchasing habits, and the like. For example, a probabilistic constraintmight be “(Probability that the viewer >30 years old)>75%.”

In addition to, or in place of, “hard” constraints in an IF X THEN Yformat, “soft” constraints may be employed, where the conditionalsprovide a weight. The item(s) returned are then those with the lowest(or highest) total weight(s). For example, consider an intermediate nodeN that is mapped to two items of supplemental content, I1 and I2. Themapping from node N to item I1 has three soft constraints: “viewer'sage >30, weight=1;” “viewer's income <$60,000, weight=4;” “viewer'sstate=Arizona, weight=−3.” The mapping from node N to item I2 also hasthree soft constraints: “viewer's age >50, weight=1;” “viewer'sincome >$100,000, weight=−2;” “time of day >12:01 am AND time of day<5:00 am, weight=−3.” To choose between the two items, the method 1500calculates the sum of the weights for each constraint. Under theknowledge that the viewer is a 40 year old in Arizona, who earns $75,000per year, the weight for the mapping to item I1 would be −2. The weightfor the mapping to item I2, at 11 am, would be 0. At 3:00 am, the weightfor the mapping to item I2 would be −9. Hence, at 11:00 am item I1 hasthe lowest weight and would be retrieved. At 3:00 am, item I2 has thelowest weight and would be retrieved instead of item I1.

FIG. 29 illustrates the use of viewer-driven constraints in the displayof supplemental content. For instance, a viewer of a media stream maysend a signal (e.g., via a remote control 2900 to a set-top box 2902)indicating that he/she is interested in viewing supplemental contentrelating to the segment identifier of the media stream currentlydepicted on his/her display device 2906. In accordance with this signal,the set-top box 2902 may read both a segment identifier identifying thecurrently displayed segment of the media stream and a viewer identifieridentifying the viewer. As illustrated, the viewer identifier is mappedto information in a database 2908 indicating that the viewer has a“high” level of income.

FIG. 30 illustrates the results of using viewer-driven constraints toselect supplemental content. Continuing the example of FIG. 29, supposethe segment identifier was mapped to supplemental content relating to apurse, a sweater and a pair of sunglasses. Further, suppose the sweateris associated (e.g., via a generic “sweater” node in an ontology) withseveral different items 3000 ₁-3000 _(n) (hereinafter collectivelyreferred to as “items 3000”) of supplemental content: a first item 3000₁ relating to a sweater to display to low income viewers, a second item3000 ₂ relating to a sweater to be displayed to middle income viewersand a third item 3000 _(n) relating to a sweater to be displayed to highincome viewers. Because the viewer's viewer identifier identifiedhim/her as being a high income viewer, the third item 3000 _(n) ofsupplemental content (relating to the sweater to be displayed to highincome viewers) is retrieved and displayed when the viewer requests moreinformation about the sweater.

The approach described herein with reference to FIGS. 14 and 15 ofmaintaining an ontology describing the interrelationships between mappedintermediate nodes, may be advantageously used in performing advanceddata mining. In a naïve data mining approach, one can calculate thenumber of viewers that expressed interest in a particular item ofsupplemental content (e.g., by “clicking through” or consummating acommercial transaction). However, such an approach misses a large amountof information that can be obtained within the present framework. Forexample, one can determine the interest shown in a particular type ofarticle by aggregating the interest shown in all subclasses andinstances contained within the ontology class that represents that typeof article.

Further, correlations can be computed. It is desirable that the ontologycontain classes and instances that allow a mapping to intermediate nodesthat not only describes the information being conveyed by a segment of amedia stream in the sense of visible articles, but also in terms ofconcepts related to the segment. For example, concepts may include the“mood” of the segment (e.g., action, car chase, romance), the backgroundmusic, a genre of the media stream (e.g., drama, horror, humor), and thelike. In embodiments where such concepts are used, a data mining processcan correlate articles, types of articles, and concepts, and thedemographics of the viewer. For example, by separately aggregatinginterest shown in the class “sports car,” between the mood concepts of“car chase” and “action,” one could determine if viewers expressinterest in sports cars more often during chase scenes than in genericaction scenes. By providing this information, if the mapped items ofsupplemental content comprise advertising, an advertiser can determinewhat types of scenes their product receives the most attention in, andtarget their advertising budget appropriately—both to the particularviewer and to demographically similar viewers. Further, movie makers andadvertisers can use the mined information to better determine the pricethat should be charged for particular product placement opportunities.

In further embodiments of the present invention, the mapping orprovision of supplemental content can be performed by the sameindividual or entity that creates or provides the media stream. Suchembodiments are especially well suited for the distribution of mediastreams over a network such as the Internet (e.g., through a servicethat hosts media streams uploaded by users). Implementing aspects of thepresent invention in connection with such a service would allow theservice to generate a robust revenue stream, which in turn would allowthe service to provide the bandwidth and storage necessary toaccommodate the volumes of multimedia content. In addition, the presentinvention would allow content creators to monetize the content they havecreated, by sharing in the revenue realized by the access provider.

FIG. 16, for example, is a flow diagram illustrating one embodiment of amethod 1600 for associating supplemental content with a media stream,according to the present invention. The method 1600 may be implemented,for instance, by a user of an access provider (e.g., a content creatoror uploader). The method 1600 is initialized at step 1602 and proceedsto step 1604, wherein a media stream (e.g., a video) is created(optionally including editing and other production steps) and loadedonto a computing device (e.g., a personal computer, a mobile telephone,a personal digital assistant or the like), in a digital format such asMPEG. FIG. 32, for example, illustrates an exemplary media stream 3200that has been loaded onto a computing device. The exemplary media stream3200 comprises a snowboarding video.

In step 1606, supplemental content is mapped to the media stream.Supplemental content may be any form of additional content, includingtext, video material, multimedia presentations, photographs, web links(URLs) and advertising copy. In one embodiment, the supplemental contentis selected from a library of content provided, for example, by theaccess provider, as described above. In another embodiment, the userprovides his/her own supplemental content. FIG. 33, for example,illustrates an exemplary item 3300 of supplemental content that may bemapped to the media stream 3200 illustrated in FIG. 32. Specifically,the item 330 of supplemental content is an advertisement/purchaseopportunity for a specific brand of snowboard pant.

In one embodiment, supplemental content is mapped with respect totemporal segments of the media stream (e.g., the segment identifieridentifying the first one minute of the media stream may be mapped to afirst item of supplemental content, while the segment identifieridentifying the second minute of the media stream is mapped to a second,different item of supplemental content). FIG. 34, for example,illustrates the exemplary media stream 3200 of FIG. 32, where the mediastream 3200 has been logically divided into temporal segments 3400₁-3400 _(n) (hereinafter collectively referred to as “segments 3400”) tofacilitate mapping of supplemental content. In another embodiment,mappings of supplemental content are made with respect to metadataincluded in the media stream encoding (e.g., segment identifiers, cuepoints, and the like).

As described in further detail below, supplemental content mapped to thesegment identifiers will be displayed either simultaneously (e.g., in aseparate frame of the same display) with the media stream or “saved” anddisplayed later, in some embodiments on a different platform in additionto or instead of the platform used to view the media stream. If thesupplemental content is to be displayed simultaneously with the mediastream, the items of supplemental content will need to appear on thescreen for a sufficient period of time so that a viewer of the mediastream has a chance to view the items. Items of supplemental content mayprovide the capability to click thereon and receive further informationrelated to the item, perhaps by opening a web browser to a relevant webpage. Such additional information could also encompass a commercialopportunity (e.g., to enable purchase of a product that was advertisedin the clicked-upon item of supplemental content). Additionalinformation may either be pre-existing information, such as ane-commerce website or web encyclopedia, or may be information createdspecifically for this purpose (e.g., a custom e-commerce store, perhapscreated to showcase a particular set of articles, for example articlescreated by the user him or herself). FIG. 35 illustrates the exemplarymedia stream 3200 of FIG. 32, wherein an item 3500 of supplementalcontent is mapped such that the item 3500 of supplemental content isdisplayed simultaneously with at least a segment of the media stream3200.

Step 1606 thereby allows a user to create a mapping from segmentidentifiers identifying segments of the media stream to one or moreitems of supplemental content. In certain embodiments, the user maycreate more than one mapping between the segment identifiers and(potentially different) items of supplemental content. For example, theuser may create different mappings according to the web site on whichthe media stream will be hosted, such that a particular media stream Swill have mapping M1 for display on web site W1 and mapping M2 fordisplay on web site W2. Such a plurality of mappings allows the mediastream S to be displayed, for example, with alternate supplementalcontent if embedded in a blog as opposed to on a web site. Later, withrespect to step 1708 of method 1700, the appropriate mapping M1 or M2will be selected for provision to a viewer in accordance with theposting location of the media stream. Further, different mappings may beprovided for display on different display devices (e.g., mapping M1 willbe displayed to users viewing on a computer, mapping M2 to users viewingon a portable device such as a telephone or personal digital assistant,and mapping M3 to users viewing on a television set).

In addition to mappings that are conditional on demographics and devicetypes, as described above, mappings may also be conditional on userbehavior, aggregate viewing statistics, or a random selection. Forexample, a plurality of mappings (or a plurality of items ofsupplemental content mapped to a particular segment identifier) may beprovided, where a choice regarding which to present is dependent on atleast one of: (1) the number of times a viewer has viewed the mediastream; (2) the number of times the media stream has been viewed intotal; (3) whether or not the viewer clicked on or otherwise expressedinterest in a previously associated item of supplemental content in themapping; (4) the average viewing time of the viewer (do they typicallyonly view the first 30 seconds of a clip?); (5) the viewer's express orlearned preferences; and (6) a random choice.

In further embodiments, a user may wish for multiple items ofsupplemental content to be simultaneously displayed with the mediastream. Step 1606 enables this functionality by allowing: (1) thecreation of multiple mappings between segment identifiers and items ofsupplemental content (where subsequent mappings may be created by adifferent user than the user/content creator who created an initialmapping); and/or (2) the mapping of a single segment identifier to aplurality of items of supplemental content (e.g., including a primaryitem of supplemental content and one or more secondary items ofsupplemental content). Secondary items of supplemental content can allowfor commentary on a primary item of supplemental content, for examplewhere a second user creates an additional mapping to comment on theassociations made in an initial mapping by an original user (e.g.,content creator). They can also allow for different users to createdifferent mappings or sets of associations, providing differentviewpoints on the same media stream subject matter.

In such an embodiment, mapping supplemental content in accordance withstep 1606 includes embedding rules within the mapped supplementalcontent. These rules allow the subsequent display of the supplementalcontent (i.e., during viewing of the media stream) to be optimized inaccordance with specified criteria (e.g., viewer demographics, time ofday or the like). In one embodiment, a rule dictates what specificinformation to display about an item of supplemental content (e.g., “IfX=TRUE, display the price of the depicted product rather than thename”). In another embodiment, a rule dictates which of two or morepotential items of supplemental content to display (e.g., “If X=TRUE,display item A; otherwise, display item B”).

Certain embodiments of the method 1600 further enable a user to arrangethe individual items of supplemental content (e.g., advertisements) intoa large set of items that will be displayed together. A user is thenable to advantageously map this set to the appropriate segmentidentifiers, instead of mapping each individual item of supplementalcontent (e.g., a pair of eyeglasses and a set of earrings would likelybe depicted in many of the same segments of a media stream, given theirphysical proximity to each other when they are worn).

In one embodiment, supplemental content is selected from a library ofitems of supplemental content (provided, for example, by the accessprovider and/or by other users), as described above. Selecting itemsfrom the library may be performed using any combination of a“drag-and-drop” tool, by checking a box, by typing in article numbers,and the like. In some embodiments, a user may specify a category ofarticles (e.g., “men's, leather belt, blue”) and allow the method 1600to propose a specific item of supplemental content according to, forexample, the most popular article for specified category, the articlefor which advertisers will paying the highest referral commissions, orother criteria of interest.

A particular library of supplemental content (e.g., pre-approvedadvertising information) may comprise a variety of information about agiven article, such as one or more of: an image of the article,information about the article, a short “commercial” about the article,and a link to immediately purchase the article. This information can beobtained from “affiliate” programs with existing online merchandisersand aggregators, and the library may further comprise customadvertising. Inclusion of item of supplemental content in the librarymay be subject to a “finder's fee” negotiated with each advertiser bythe access provider. For example, a certain seller's affiliates mayreceive as much as x % of the purchase price of any article sold to acustomer, when the customer was directed to the seller's site by theseller's affiliate's site.

Embodiments of the invention allow third parties (all, or an authorizedgroup of users) to map supplemental content to segment identifiersidentifying segments of a given media stream, providing asyndication-like system. Revenue from syndicated media streams can besplit between the originator of the media stream (i.e., the contentcreator) and the syndicator who mapped the supplemental content. In thisway, the value of the media stream is increased because the task ofrealizing value is spread across a number of different people (e.g., thecontent creator may have no interest in creating mappings, whereasothers are very interested but lack suitable media stream content).

In one embodiment, third-party syndicators are provided with ratings,for example according to their success in monetizing media streamcontent or according to other factors. By monitoring viewer response,statistics can be generated to indicate that a particular syndicatorgenerates high viewer interest in their association mapping for this“type” of media stream. Media stream “type” in this context could bebased, for example, on the content tags (e.g., Elvis) or on thedemographics of the people watching the media stream when it is firstlaunched (e.g., a video clip being watched primarily by teenage girls).A marketplace is therefore enabled to match media streamcreators/uploaders with syndicators. Embodiments of the presentinvention further allow the media stream creator to control thesyndication of his or her media stream, for example by setting certainminimum performance metrics such that if the mapping provided by asyndicator does not generate sufficient results, the mapping will beremoved from public availability. In one embodiment, performance metricsinclude absolute thresholds (e.g., “must generate at least 1,000 viewerresponses in the first 10,000 views”) or relative metrics for mediastreams with more than one syndicator mapping (e.g., “only the top threesyndicator mappings are retained after forty-eight hours). Newsyndicator mappings could be added for a trial period to compete in therelative metrics, either on a random basis or on a percentage basis(e.g., 90% of views will receive the currently best performing mapping,and 10% will receive a new syndicator mapping), and after a certainperiod of time the performances of the various syndicator mappings arecompared, with the winner becoming the new best performing mapping.

In one embodiment, the access provider facilitates a beneficial divisionof labor by enabling individual syndicators to form and work together insyndication teams. One benefit of such teams is that they can bestructured to bolster the standard viewer-driven optimization processfor determining the most compelling items of supplemental content byproviding a “pre-screen” in the form of a team pre-approval process. Forexample, each member of the team who wishes to provide an item ofsupplemental content may be required to submit the item of supplementalcontent to the team for review. In this approval process, the variousteam members may provide commentary (e.g., through written comment orverbally such as by voice over internet protocol or VoIP) by modifyingthe submitted item of supplemental content (potentially through a sharedworkspace as described earlier herein), or even by proposing alternativeitems of supplemental content. The team may require that submitted itemsof supplemental content reach a certain threshold of acceptance amongthe team members before the items of supplemental content can besubmitted on behalf of the team. For example, a team may require that anitem of supplemental content receive a certain percentage of “Yes” votesfrom team members (potentially with additional criteria such asrequiring that the total number of votes cast must be at least a minimumpercentage of team members). Such rules may require that voting beclosed only after the item of supplemental content has been presented tothe team for a set period of time, in order to ensure sufficientopportunity for review. This type of process will help to ensure thathigher quality items of supplemental content are submitted and also willtend to limit the number of items of supplemental content that areoffered, thereby helping avoid “overwhelming” automated optimizationprocesses. The access provider may also provide access point“multipliers” of various sorts to encourage specific types of behavior(e.g., “double points” for being the first user or collaborator to mapitems of supplemental content to a given media stream, having a items ofsupplemental content “win” an optimization process, etc.).

The access provider, or even the creator of the media stream, may beenabled to impose certain requirements for or limitations on syndicatorsubmissions. For example, a content creator may require that syndicationteams follow certain guidelines (e.g., minimum voting requirements) orhave certain characteristics (e.g., teams comprised of more than 10members, where teams holding at least X access points will be permittedto submit items of supplemental content to the queue once the mediastream has reached a certain level of popularity). In one embodiment,syndicators may be permitted to “buy around” such limitations byagreeing to a pay fee (or, for example, by agreeing to receive a lesserpercentage commission from generated revenue) in exchange for gainingexemption for the standard submission rules.

To facilitate team formation, the access provider may provide aselection of various types of team structures around which teams may beformed. For example, one team structure may be governed by completemajority rule, with all decisions (e.g., whether or not to contribute aparticular item of supplemental content, how to divide commissions,etc.) open to a community vote. Another example of team structure may becontrolled by a board of directors who make all decisions, or astructure that is led by a single individual, such that most members mayonly submit items of supplemental content for consideration and but maynot be granted a vote. Various structures may be combined (e.g., a boardof directors set to make all decisions, except that the decision tochange any commission procedures must be made by majority rule).

FIG. 39 illustrates the dynamics of an exemplary syndication team,according to the present invention. Specifically, FIG. 39 illustrates anexemplary display 3900 for conveying syndication team data. For example,the display 3900 may comprise a first area 3902 for displaying the mediastream under analysis (or segments thereof) and a second area 3904 fordisplaying an item of supplemental content associated with a currentlydisplayed segment of the media stream (i.e., currently displayed in thefirst area 3902 of the display 3900). Additionally, the display 3900includes a suggestion area 3906 in which team members can suggestalternate items of supplemental content for association with thecurrently displayed segment of the media stream and a commentary area3908 for commenting on the suggestions in the suggestion area 3906.

As illustrated, each suggestion in the suggestion area 3906 includes asuggested item of supplemental content (e.g., “It's the pants” versus“Gotta wear shades!”) and a vote counter that tracks the number of teammembers who have voted for selection of the suggested item ofsupplemental content. As described above, the suggested item ofsupplemental content garnering the greatest number of votes is typicallyselected for the mapping.

In addition, the display 3900 may include a plurality of avatars 3910 orother icons or identifiers, one for each member of the syndication team.

The access provider may wish to encourage team formation, since poolingthe perspectives and knowledge of various team members (e.g., byselecting items of supplemental content by majority vote) and/orspecializing in certain types of media streams and/or items ofsupplemental content (e.g., videos about soccer or items of supplementalcontent relating to high-fashion advertising) may ultimately producemore compelling mappings. In order to encourage syndicators to formteams, the access provider may offer to pay team-submitted items ofsupplemental content a higher percentage of commissions. In addition,the access provider may allow teams to aggregate the “access points” (asdescribed above) earned by team members so as to gain faster access tothe most popular media streams. In the point aggregation approach, it islikely that the access provider would impose a higher access pointthreshold on teams with respect to each category of media stream;however, the threshold would not be so high as to discourage individualsyndicators to form teams. For example, the access provider may requirethat in order to earn the right to submit items of supplemental contentfor the top 100 most popular media streams currently hosted by theservice, an individual syndicator may be required to hold at least Xnumber of access “points,” teams of five people or fewer must have atleast 3X points, teams of six to twenty people must have 10X points andteams over twenty must have at least 20X points.

In one embodiment, the access provider or a third party creates a“marketplace” in which various users and teams may buy, sell, auction,barter for, or otherwise exchange “access points” with others. Forexample, a content creator who has provided a relatively small number ofmedia streams may gain access points that he/she has no use for, giventhat he/she does not plan to provide any additional media streams oritems of supplemental content. This content creator may sell or auctionhis/her access points to other users, such as to a team of syndicatorsthat wishes to reach the next level of media stream access. The host forthe auction may charge a fee for such transactions. The access providermay also impose an exchange rate, so as to encourage individual users toretain their access points and, hopefully, encourage these users toparticipate actively in the community. For example, the access providermay require that access points sold by one user to another individualuser are reduced in half by the transaction (i.e., ten access pointsbecomes five access points upon transfer), and may potentially impose adifferent exchange rate for points transferred to teams (e.g., tenaccess points becomes two access points), so that teams are less likelyto dominate the entire community.

Once a media stream is opened for syndication, the content creator orothers may be permitted to delete or block certain items of supplementalcontent offered by syndicators. This is an important feature, given thatfewer content creators will want to open their media streams tosyndicators if doing so causes them to lose all control over their mediastreams. However, assuming that content creators would receive a smallercommission on items of supplemental content proposed by syndicators(i.e., there is likely a financial incentive for the content creator toopen the primary media stream to syndicators, as well as an incentivefor syndicators to offer items of supplemental content), the accessprovider may require that once a content creator opens a media stream tosyndication, the content creator will thereafter receive no commissionhigher than he/she would receive from a syndicated item of supplementalcontent. Otherwise, content creators would have an incentive to observewhich items of supplemental content proposed by syndicators weresuccessful and then replace lower commission-paying items ofsupplemental content with virtually identical supplemental content thatdoes not entail sharing commissions with a collaborator.

In some embodiments, step 1606 is configured with constraints regardingthe permitted mappings to supplemental content (e.g., the minimum andmaximum time for which a particular item of supplemental content may bedisplayed, or the minimum and maximum number of commercial versusnon-commercial supplemental content mappings). In some embodiments,synchronization may be constrained (perhaps with user consent) such thatgaps of M seconds must exist between synchronizations made by the user.The intervening M seconds may then be automatically filled bysupplemental content automatically selected by the access provider, andrevenue from these intervening items potentially shared with the user.

At step 1608, the user uploads the media stream and at least one mapping(associating the media stream with supplemental content) to the accessprovider. In one embodiment, step 1608 includes requiring the user tosign a statement confirming that he or she is also the content creatorand owner of the copyrights in the uploaded media stream. In oneembodiment, the user is also provided with the options of providing“tags” that describe the media stream with certain keywords, assigningthe media stream to one or more categories by genre (e.g., comedy),length, type of creator (e.g., professional), and the like.

Some embodiments of step 1608 will allow the content creator to definethe look and feel of the media player that will display their mediastream. For example, the creator of a nature-related video might selecta green, leafy media player motif that features wood textured play,pause, and fast-forward buttons. A hip-hop oriented creator might electa more contemporary “bling” player.

The method 1600 then terminates in step 1610.

In some embodiments of method 1600, a local application on theuser/content creator's computing device is used to implement the method1600. In other embodiments, a web-based application is used to implementthe method 1600. In such web-based embodiments, the step 1608 may beperformed in two parts, where the uploading of the media stream occursbefore the associating step 1606, and the uploading of associationsfollows step 1608 as a separate step. In further embodiments, parts ofthe method 1600 are performed locally and other parts are performedremotely on a networked computing device.

Embodiments of the method 1600 incorporate mechanisms for reducing theuploading of pirated content (i.e., copyrighted content uploaded by anindividual other than the copyright holder). For example, when an accessprovider is notified (e.g., by the copyright holder) that a given mediastream may be violating certain laws or regulations, the access providermay find it necessary or useful to automate certain processes thatderive from this situation. In one embodiment, if the access providerfinds it necessary to remove a media stream from its offerings, theremoval may trigger a number of related actions, including one or moreof: temporarily banning the user who provided the media stream fromusing service provided by the access provider, reporting the user whoprovided the media stream to an account administrator and notifying theuser who provided the media stream and any collaborators and whocontributed thereto (e.g., by assisting in the creation of a mapping ofsupplemental content or by actually providing the supplemental content)of the removal. Furthermore, a summary of financial informationgenerated by the media stream (e.g., monetization until the point ofremoval) may be generated and delivered to various stakeholders, and anyfunds currently held by the access provider and owed to the user whoprovided the media stream may be placed on hold.

In one embodiment, supplemental content associated with a media streamin accordance with the method 1600 is embedded with the media streamsuch that any monetization (e.g., click-throughs) associated with themedia stream reverts to the content creator, regardless of theplatform(s) to which the media stream may be exported. FIG. 46, forexample, illustrates the use of an initial annotated media stream 4600across various platforms including a social networking website 4602, aweb log (“blog”) 4604 and a portable media player 4608. As illustrated,regardless of the platform 4602, 4604 or 4606 to which the initial mediastream 4600 is exported, any supplemental content associated therewithremains associated.

FIG. 17 is a flow diagram illustrating one embodiment of a method 1700for providing a service for providing access to a media stream with atleast one mapping to supplemental content. The method 1700 may beimplemented, for example, by a web site that hosts media streamsuploaded by users/content creators. The method 1700 is initialized atstep 1702 and proceeds to step 1704, wherein a media stream and one ormore mappings associating the media stream with supplemental content arereceived by the service provider.

In some embodiments, when the media stream is uploaded, the method 1700is enabled to obtain both the program identifier (e.g., identifying theparticular media stream) and a segment identifier associated with eachsegment of the media stream. The method 1700 may then additionallyreceive from the user/content creator various forms of metadataassociated with the segment identifiers such as scene changes, “mood”changes, shot changes, etc. and allow those inputs to be stored forlater use by the user or by the access provider.

In optional step 1704 (illustrated in phantom), the media stream istranscoded from the source format into the preferred format of thehosting service, if required. For example, the hosting service mayprefer to store all media streams in Flash format (a video formatsupplied by Adobe, Inc., of San Jose, Calif.), wherein the method 1700would, in step 1704, transcode an uploaded MPEG video into the Flashformat.

In step 1706, the method 1700 makes the media stream available toviewers, and transmits the media stream, association mapping(s),supplemental content, and a viewing program to the viewer, on demand. Inone embodiment, this is accomplished by posting the media stream andother material on a web site, embodied in a single application (e.g., aFlash application) that provides a simple user interface for playing,pausing, seeking, and stopping the media stream, adjusting the volume ofthe audio track, and displaying the associated supplemental content inaccordance with the mapping.

In certain embodiments, the access provider may allow operators of othersites to link to a particular media stream and include that media streamon the other sites. In one embodiment, this is configured such that themedia stream is only linked along with the mechanisms for displaying themapped supplemental content, ensuring that viewing of the media streamon the third party website will include display of the mappedsupplemental content. In one embodiment, a revenue model for thisembodiment involves a negotiated sharing of revenue between the thirdparty host and the original access provider for any viewer purchases orother revenue-generating click through activity.

If multiple mappings to supplemental content are available for aparticular media stream (e.g., the media stream was syndicated forothers to create mappings), many mappings may have been created, buttypically only one mapping will be displayed with the media stream at atime. As discussed earlier, some embodiments may tag particular mappingsaccording to viewer device type, demographics, and so on. In suchembodiments, the mapping is chosen, or a subset of available mappingsselected, based on such criteria, in accordance with data provided bythe viewer's device (e.g., as part of an HTTP request header, IP addressto geographical mapping, and the like), or by preferences for types ofmappings set by a viewer. If this does not yield a single mapping, themethod 1700 may choose which mapping to present on a down-selectionbasis. For example, assume that a given media stream is syndicated andfive different mappings are created. Those five mappings could be shownequally to the first N number of viewers, with the least popular(aggregate less clicks, or lower revenue-generating) mapping being downselected at each subsequent N number of views, until only the mostpopular mapping is shown. By monitoring viewer selections, the method1700 may further determine the better mappings to be shown to particulardemographics, even though the mappings themselves did not includedemographic targeting information. For example, user responses mightsuggest that a given mapping is popular with teenagers, in which casethe method 1700 would provide the mapping to teenagers, while adifferent mapping may be shown to thirty-something's.

At step 1710, a signal is received from a viewer, who is watching aparticular media stream and mapping and who selects (e.g., clicks on) anitem of associated supplemental content (e.g., advertisement).

At step 1712, the method 1700 provides supplemental content to theviewer (e.g., a web page configured to provide fulfillment of a purchasefor the article depicted in the associated supplemental content).

As described above, in certain embodiments when the provision ofsupplemental content results in the viewer making a purchase orperforming other activity (e.g., click through activity) that accruesrevenue to the access provider, the access provider will pass along aportion of the revenue that it receives to the user/content creator whoposted the media stream whose mapping initiated the revenue as acommission. The commission paid to posting user may be a percentagecalculated from the specific sales consummated.

In certain embodiments, the access provider simply tracks the number ofclick-throughs that result from a specific media stream and divides thatnumber by the total number of click-throughs generated across all mediastreams hosted by the access provider during a given period of time,yielding a percentage of click-throughs attributable to the specificmedia stream. Then, the total pool of commissions that are available tobe distributed by the access provider for such period may be allocatedto the posting users/content creators according to this calculatedpercentage share. The access provider might choose to provide agraduated or hierarchical form of commissions to encourage specificactivities. For example, the access provider might pay a highercommission to posting users/content creators who provide three or moremedia streams during a given period that attract at least apredetermined threshold number of clicks. Also, as an alternative topaying a cash commission to posting users/content creators, the accessprovider might provide some form of “credit” or “coupon” that could beredeemed for various goods or services provided by the access provideror third parties.

In certain embodiments, the access provider may wish to provideincentives for users to include non-commercial items of supplementalcontent, so as to counter the naïve belief that monetization may beenhanced simply by increasing the volume of advertising-relatedsupplemental content presented to a viewer. This may be because a higherratio of non-commercial to commercial items of supplemental contentincreases viewer satisfaction and/or leads to greater overallmonetization. For example, the access provider may wish for users todeploy items of supplemental content that potentially direct viewers toother portions of the access provider site such as special “branded”pages, pages featuring “pay-per-view” videos, and the like. To encouragesuch deployment, the access provider may choose to pay a fee to the userfor every time such a navigation-oriented item of supplemental contentis either viewed or clicked upon by a viewer. Similarly, the accessprovider could link payment to the viewer's subsequent behavior inresponse to such link (e.g., pay the user a commission only with respectto each viewer who selects to view a pay-per-view video), in what may betermed a “cost-per-action” or CPA model.

The access provider may also wish to provide incentives for contentcreators to provide intra-media stream content tagging (i.e.,identifying various categories of content within a given media stream toimprove viewer search). To induce this behavior, the access provider mayprovide a fee or commission for each time that an intra-media stream orother form of tagging results in a viewer actually locating and/orviewing a media stream. For example, if a content creator has includedan intra-media stream content tag for “Eiffel Tower” and a viewerconducting a search is presented with the thumbnail pertaining to thiscontent tag and/or the viewer clicks and views the associated mediastream, the content creator may receive a small commission or othercompensation for any one or all of the various aspects that this/hertagging enabled.

In all cases in which the access provider may choose to share revenue,it may be useful to set certain threshold levels for when payments arecalculated, reported, paid, and the like. For example, given theadministrative cost and burden of handling a large volume of very smallmonetary amounts, the access provider may decide to display all suchinformation but refrain from making any payments until a certainthreshold is reached, such as a given item of supplemental content isviewed by more than X number of viewers or a given user is entitled toreceive at least ten dollars. The access provider may also impose suchthresholds for the purpose of limiting the ability of others tomanipulate the payment system by providing fraudulent click-throughs.For example, the access provider may decide to pay a relatively low X %commission for the first Y viewer interactions with an item ofsupplemental content, and 20X % for the next Z viewer interactions,which represents a threshold of interactions that would be verydifficult without a large number of people participating in suchmanipulative enterprise. Such a tiered commission system would provideadded incentives for users to create highly-compelling items ofsupplemental content. As discussed above, however, the access providermay determine that providing lower commissions for more popular mediastreams may be beneficial in terms of encouraging greater syndicatoremphasis on providing items of supplemental content for less popularmedia streams.

In one embodiment, the access provider may provide a commission to theuser who posted a library entry or a component thereof in thesupplemental content library that was utilized by another user increating an item of supplemental content and/or that generated revenue.This would encourage contributions to the library. Certain thresholdsmay also be implemented in this context, so that contributions areremunerated only if they generate a very high level of revenue. Theaccess provider may not wish to discourage users from accessing thelibrary, but at the same time, the access provider may wish to induceusers to make highly compelling contributions.

Advertisers who wish to select a “cost per impression” (CPM) model in aparticular context (i.e., the advertiser will pay the access providerfor advertisements that are viewed, but not necessarily clicked on) maybe provided with a tiered pricing model, such that items of supplementalcontent mapped to earlier segments of a media stream are priced higherthan items of supplemental content mapped to later segments of the mediastream. This model considers the fact that viewers may be more likely torespond to items of supplemental content associated with the earlyseconds or minutes of a media stream and/or that many viewers will notwatch an entire media stream (and, therefore, may never actually view alater-mapped item of supplemental content).

As discussed above, any of the revenue sharing models could be enabledto share revenue across a “team” of people who provide relevant items ofsupplemental content. In this manner, users specializing in one aspectof the process (e.g., selecting a product to feature) may couple theircontributions with users specializing in another aspect (e.g., creatingforceful graphics) so that the overall end-product is more compelling toviewers and therefore generates higher revenue. For example, withrespect to a single item of supplemental content mapped to a singlemedia stream, it may be the case that a certain aggregate commissionpercentage of X is paid by the access provider to all of thecontributors to the revenue opportunity. Thus, a portion of commission Xmay go to, among others, individuals or groups that have contributed thefollowing aspects of the revenue generation process: uploading the mediastream; intra-media stream content tagging; contributing certaincomponents that appeared in the item of supplemental content (e.g., aparticular graphical image in a composite item of supplemental content);providing the specific item of supplemental content; and hosting a linkthat directed the viewer to view the media stream. In such a context,the individual or group that provided the media stream would potentiallyreceive a commission with respect to most or all activities relating tosuch media stream, whereas the individual or group providing aparticular item of supplemental content would receive a commission onlywith respect to revenue derived from the item of supplemental content.In all of these cases, revenue may be paid by the access providerservice into a “team” account.

To facilitate the revenue tracking and payments systems described above,the access provider may embed within a given item (e.g., an annotatedmedia stream, an item of supplemental content, a library entry) anidentifier that identifies the contributor of such item. Thus, a givenannotated media stream may include identifiers for a potentially largenumber of contributors. Each contributor identifier may be linked to afile of information about the identified contributor to facilitatepayment, such as the contributor's social security number, desiredmechanism for payment, and the like.

As described above, the access provider may allow various teams orgroups of individuals to participate in the creation of any of thevarious aspects of an annotated media stream. To encourage the formationof teams, the access provider may provide a mechanism for each team topersonally determine how the revenue accruing to the team will be sharedamong the various team members. For example, the access provider mayallow a team to decide if all members share equally in all revenue, orwhether a certain portion of revenue will be allocated to key members ofthe team (e.g., the member who contributed a particular item ofsupplemental content, media stream or the like or members that havereached a certain threshold of seniority, performance, or the like) withthe remainder provided to be split among the team as a whole.

In one embodiment, the method 1700 tracks the responses of the viewer,including the signal indicating interest in an item of supplementalcontent received at step 1710, and the user's behavior in response toany additional information provided in step 1712 (e.g., consummating acommercial transaction). The tracked data may be aggregated to providestatistics regarding each media stream, each mapping, and each item ofsupplemental information, to enable users of the system to determinepopular media streams and successful mappings. This response trackinginformation could be provided to one or more stakeholders (e.g., thecontent creator or all sellers whose articles are depicted in the mediastream) based on one or more criteria (e.g., syndicators gain access to“professional level” statistics only after a history of success).Multi-tier reporting regimes could be implemented such that additionaldata requires either payment or attainment of certain performance levels(e.g., volume of clicks or transactions). Conversely, it might be useful(perhaps at the uploader's option, or as a “professional” mode of theplayer) to allow the display of viewer response to a given mapping orindividual associated of supplemental content on the video playeritself. This would provide viewers with information as to what their“peers” are interested in, and thereby provide feedback on how togenerate a successful mapping.

The method 1700 then terminates in step 1714.

One of the advantages of the present invention is that, as describedabove, previously unrelated individuals may be brought together by theprospect of contributing to a common experience (i.e., improving thecreative value of a given media stream) and/or by the prospect ofearning money. The access provider will benefit financially by creatingmore compelling viewer experiences due to the fact that higher qualityexperiences attract more viewers and more relevant supplemental contentwill increase viewer response to a particular media stream (and/or thesupplemental content mapped thereto). In order to bolster the creationof such content, the access provider may wish to provide a directmonetary incentive to all individuals who add value.

Moreover, supplemental content is provided to viewers of a media streamwithout requiring any explicit involvement or instruction from theviewer. However, the supplemental content may be easily ignored by aviewer who is not interested in viewing the supplemental content; thus,the mapping of supplemental content does create any undue intrusion onthe viewing experience. It is to be noted, however, that viewerinteraction may enhance the viewer experience by allowing thesupplemental content provided to a given viewer to be tailored tohis/her preferences. For example, the viewer may select or indicate atype of supplemental content in which he/she is most interested (e.g.,fashion, recipes, music, movie plots, director's cut videos or thelike), or such selections may be learned.

FIG. 18 is a flow diagram illustrating one embodiment of a method 1800for displaying a media stream with mapped supplemental content to aviewer. The method 1800 may be implemented, for example, by a service(e.g., web site) that hosts and displays media streams, on demand.

The method 1800 is initiated at step 1802 and proceeds to step 1804,wherein a media stream is displayed to a viewer (e.g., at the request ofthe viewer). In some embodiments, the method 1800 enables viewers tosearch for media streams containing specific items. For example, aviewer might use a key word search to find all media streams thatfeature an advertisement for the key word “Brady Bunch.” In oneembodiment, the method 1800 supplies information to the remote computerproviding the access provider regarding the viewer's display device typeand capability (e.g., screen resolution, computational capacity, networkbandwidth or the like). Further information may be supplied regardingdemographics or geographies of the viewer (e.g., his or her location).Further information may be supplied based on viewer-selected preferencesfor particular types of mapping (e.g., that the viewer prefers “humor”supplemental content, or likes supplemental content that containsdiscount coupons).

In certain embodiments, when a media stream is viewed in full-screenmode or on a small screen, the displaying allows the viewer to elect(e.g., by pressing a button) to send information regarding supplementalcontent mapped to the media stream to a website for later viewing, or tosimply send the information back to the screen from which he selectedthe media stream.

At step 1804, supplemental content that is mapped to the media stream isdisplayed to the viewer. In one embodiment, the supplemental informationis displayed to the viewer in a frame or display location adjacent tothe playing media stream (such that the viewer can simultaneously viewboth the displayed media stream and the supplemental content), and notas an overlay or other intrusive interface device. An exemplary displaydevice 1900 is depicted in FIGS. 19A-19D, which respectively illustratevarious formats for displaying supplemental content. Display device 1900comprises a main display area or pane 1902 for displaying the mediastream, which, in the exemplary case of FIGS. 19A-19D, is a video of aschool dance. At least one supplemental pane 1904 ₁-1904 _(n)(hereinafter collectively referred to as “supplemental panes 1904”) isprovided for synchronized display of the supplemental content associatedwith the video, which in the exemplary case FIGS. 19A-19D is presentlyan advertisement for a music CD, perhaps of the music presently playingat the school dance at the present time in the video. Media streamscontrols 1906 are provided for the standard operations of playing,pausing, and seeking the media stream content, and the present timeposition within the displayed media stream is shown in counter 1908,along with the total length of the media stream.

As illustrated in FIG. 19, the display of the supplemental content issynchronized with the display of the media stream, in accordance withthe association information. For example, assume a first item ofsupplemental content is mapped to the segment identifier identifying thesegment that spans time t=10 to time t=40, and a second item ofsupplemental content is mapped to the segment identifier identifying thesegment that spans time t=20 to time t=30. At time t=10, the first itemof supplemental content is displayed to the viewer. At time t=20, boththe first and second items of supplemental content are displayed. Attime t=30, only the first item of supplemental content is displayed, andat time t=40 neither the first nor second items of supplemental contentare displayed. No specific viewer intervention is required to activatethe display of the supplemental content (i.e., the supplemental contentis displayed automatically in accordance with the synchronizationestablished by the posting user/content creator, taking into account anypausing, seeking and stopping of the media stream by the viewer). Incertain embodiments, more than one set of supplemental content may bedisplayed simultaneously (e.g., by adding additional supplementalpanes), and all displayed supplemental content is similarly synchronizedwith the media stream.

Where the media stream player is embedded in another content pane (e.g.,a web page, a blog or the like), other content displayed on thecontaining pane may be selected in accordance with the media stream orassociated supplemental content. For example, banner advertisementsdisplayed on the containing pane may be selected based on keywordsassociated with the media stream, based on the target demographic of themedia stream, based on the supplemental content mapped to the mediastream, based on the position of an item of supplemental content in ahierarchy of a library, or based on viewer information collected by thesystem to determine which mapping to use for a particular viewer (e.g.,demographic, geographic or device information, or prior interactionswith supplemental content). For instance, FIG. 45 illustrates anexemplary display 4500 in which keywords in the supplemental content4502 associated with a media stream 4504 are used to generate relatedbanner advertising 4506.

In one embodiment, each time that a media stream is accessed forviewing, the method 1800 may gather various relevant data (e.g., if themapping(s) for the media stream includes placeholders; the determinersthat may be embedded in various alternative items of supplementalcontent; the demographic information provided by the website from whichthe viewer is accessing the media stream, etc.) and, either prior todisplaying the media stream or in near real time with display of themedia stream, determine the specific items of supplemental content to bedisplayed. Once this data is gathered, the method 1800 may stream themedia stream and then simultaneously stream the mapped items ofsupplemental content into a single stream or application. Alternatively,the method 1800 may automatically determine all of the differentpossible combinations of supplemental content (e.g., all possiblecombinations of the supplemental content that are linked to placeholder)and create a separate data stream for each combination. Each separatedata stream so created may be combined with the media stream ormaintained separately from the media stream. In this embodiment, whenthe viewer accesses the media stream, the method 1800 determines whichversion of the supplemental content mapping (or which version of thecombined mapping and media stream) that is presented to the viewer.

In one embodiment, the mapping is entirely independent of the mediastream that is presented to the viewer, without any direct connectionwith or access to data from the media stream. For example, the mediastream may be offered by a first access provider, and a second accessprovider that offers the mapping may make the mapping available to anythird party who requests the mapping (e.g., by providing the“identifier” of the media stream that the second access providerutilizes to select the mapping). To ensure synchronicity with the mediastream, it may be the case that the media stream and the mapping arepresented to the viewer via two separate player devices (e.g., with theplayer that is playing the supplemental content linked to the playerthat is playing the media stream such that, when commands like “fastforward” and “pause” are given by the viewer to the media stream player,the same commands are mirrored in the player displaying the supplementalcontent. This functionality will allow the supplemental content to be insync with the media stream, but allows the supplemental content to behosted by a service other than the service hosting the media stream.This may be advantageous, because the bandwidth costs or otherrequirements for hosting media streams may differ from the requirementsfor hosting mappings, and, therefore, there may be advantages toseparating these two functions.

In certain embodiments, the access provider may allow operators of otherwebsites to link to a particular media stream and to include the mediastream on their other websites. In one embodiment, this is configuredsuch that the media stream is only linked along with the mechanisms fordisplaying the associated items of supplemental content, ensuring thatviewing of the media stream on the third party websites will includedisplay of the items of supplemental content. In such a context, allviewer interaction with the supplemental content (e.g., a viewerclicking on a given item of supplemental content) may be transferredback to the access provider. Alternatively, the viewer interactions areenabled on the third party websites, potentially so that a record of theinteractions is shared with the access provider. The revenue modelsassociated with these embodiments may involve a negotiated sharing ofrevenue between the third party host and the primary access provider forany viewer purchases or other revenue-generating click-through activity.

Once the items of supplemental content are mapped to a media stream, themethod 1800 may create a log listing of the particular library entriesand/or components thereof that are presented in the media stream (aswell as the temporal segments of the media stream in which the libraryentries appear). When the media stream is activated by a viewer, themethod 1800 can thereby record all of the items of supplemental contentthat were presented to the viewer, and not merely those items ofsupplemental content that were “clicked” by the viewer. This will aidthe access provider in charging advertisers or others parties forpresenting information in addition to facilitating situations in whichviewers take specific action with respect to the presentation ofinformation (such as clicking on an item of supplemental content toaccess additional information).

Referring back to FIG. 18, at step 1808, the method 1800 receives asignal from the viewer indicating interest in a particular item ofassociated supplemental content. The signal is passed to the accessprovider (or other third party) for fulfillment of the request, and atstep 1810, further information regarding the signaled request isdisplayed to the user (as described from the viewpoint of the accessprovider earlier in relation to steps 1710 and 1712 of method 1700).

The method 1800 then terminates in step 1814.

The methods herein described may thus be implemented to transform asimple media stream access provider into one that enables additionaluser creativity, and rewards the posting user/content creator for theactivities induced in the viewers. The access provider and advertisersor other providers of supplemental content may also derive revenue fromsuch a system.

It may often be the case that a particular media stream has a “lifespan”, but may also be “reborn.” For example, a cute puppy video is bothvery compelling and very forgettable (and/or supports repeated viewing).Thus, an operator of an access provider may automatically, or at theposting user's request, “shelve” a media stream for a set period of timeand have it reposted on the site at a later date that is eitherpredetermined (i.e., the media stream is released automatically) or uponaction by the posting user. The access provider might also monitor“viewing patterns” for specific media streams and send out automaticemails to the posting user suggesting that they might wish to removetheir media stream for a period of time. The access provider might allowsuch media streams to still be available to viewers utilizing one formof search (e.g., the puppy clip would appear in response to a specifickeyword search) but would not appear in response to certain other searchmethodologies (e.g., it would not appear on the “most watched” lists).This would help the access provider to avoid appearing “stale” by havingthe same media streams appear at the top of lists. One way to do thiswould be to use an algorithm that combined the “most watched” with aconcept of “most recently watched.” This approach would, for example,result in a World Series-related video being removed from themost-watched category (even though it was widely viewed during a shortperiod of time) in June in favor of a “fresher”, but less watched, NBAfinals video.

In some embodiments, an access provider may allow advertisers or otherproviders of supplemental content to “pull” (i.e., remove or block thepresentation of) their items of supplemental content if they are notpleased with a particular association. For example, an automobilemanufacturer might not want to allow their SUV advertisement to beassociated with a video about global warming. To enable this, a searchfunction may be provided to determine all media streams with which aparticular item of supplemental content is associated, and the abilityto remove an association is subject to a security-enabled process (e.g.,a password system) so that only the rightful provider has this ability.For example, FIG. 41 illustrates an exemplary display 4100 that may bepresented to an advertiser to allow search of media streams associatedwith a given item of supplemental content. FIG. 42 illustrates anexemplary display 4200 in which results of a search such as that enabledby the display 4100 of FIG. 41 are presented. FIG. 43 illustrates anexemplary display 4300 in which an advertiser is enabled to blockassociation of an item of supplemental content with a given mediastream.

In an embodiment using libraries of supplemental content, portions ofthe library may be marked as appropriate only for association withparticular types of media streams. This marking may be advisory innature, or may be enforced by the system such that association with themarked content will not be permitted. Content may also be marked withparticular viewer demographics (e.g., such that content directed to, orunsuitable for, particular age ranges is only, or not, available formappings provided to such demographic). Further, during presentation ofa media stream and its associated supplemental content, a method may beprovided for the viewer to flag potentially improper or undesirableassociations for human review, either on the initial flagging by aviewer or when a certain absolute or percentage number of viewers flagthe association.

In certain embodiments, where multiple mappings to supplemental contentare available, a rotation method may be employed to select mappings forpresentation to a viewer. Referring, for example to FIG. 36, whichillustrates three items 3600 ₁-3600 ₃ (hereinafter collectively referredto as “items 3600”) of similar supplemental content for potentialmapping to a segment identifier associated with a segment of a mediastream 3602, if three different mappings M1, M2, and M3, are availablefor a particular media stream, the mappings may be rotated such that thefirst viewer of the media stream receives mapping M1, the second viewerof the media stream receives mapping M2, and the third viewer of themedia stream receives mapping M3, each thereby receiving potentiallydifferent items of supplemental content at different synchronizationtimes in accordance with the received mapping M1, M2 or M3. A fourthviewer of the media stream will then receive mapping M1, and so on. Theprovided mappings may then be narrowed, such that after N views of eachmapping, the mapping yielding the highest viewer interest (e.g.,measured by number of clicks on items of supplemental content, thenumber or value of commercial transactions consummated as a result,etc.) is chosen as the only mapping for subsequent provision inconnection with the media stream. For example, referring to FIG. 37,which illustrates click-through counters 3700 ₁-3700 ₃ (hereinaftercollectively referred to as “click-through counters 3700”) for eachrespective item of rotated supplemental content illustrated in FIG. 36,item 3600 ₂ generated the highest number of click-throughs among thethree items 3600 out of 383 click-throughs. Thus, item 3600 ₂ isselected for display with the relevant segment of the media stream 3602.

As illustrated in FIG. 38, which illustrates click-through counters 3800₁-3800 ₃ (hereinafter collectively referred to as “click-throughcounters 3800”) for different respective items 3802 ₁-3802 ₃(hereinafter collectively referred to as “items counters 3802”) ofrotated supplemental content, a similar process can be implemented inwhich collaborators suggest various items of supplemental content forpotential mapping to the media stream 3602. In the exemplaryillustration, a first collaborator has suggested an item 3802 ₁ relatingto a pair of ski goggles; a second collaborator has suggested an item3802 ₂ relating to a ski hat; and a third collaborator has suggested anitem 3802 ₃ relating to a pair of ski pants. The item 3802 ofsupplemental content receiving the highest click-through rate (in thiscase, the third item 3802 ₃) after a number of rotations is selected forfinal mapping.

Alternatively, the mapping with the lowest viewer interest may beremoved from the rotation, and the surviving mappings rotated untilthere is only one mapping remaining after subsequent rounds of rotationand removal. Rotation and removal/selection may be performed for allviewers of a media stream, or just for specific demographics (e.g., onemapping for male viewers and a rotation approach to select among aplurality of available mappings for female viewers).

Where multiple mappings are provided for a particular media stream, eachmapping to a different set of supplemental content, a user may wish tocompare the success of particular associations for particular segmentsof the media stream. In cases where the associations are not preciselysynchronized at the same time in each mapping, the comparison operationmay track user interest within a certain time window before and afterthe presentation of a particular item of supplemental content, toharmonize the lack of exact synchronization.

Traditional media stream access providers provide rankings of mediastreams in terms of “most watched,” “highest rated,” “most commentedupon,” and the like, in order to provide viewers a key into potentiallyinteresting and entertaining content. In embodiments of the presentinvention where data is available regarding viewer interest in thesupplemental content associated with a particular media stream, the datamay be used to provide a ranking system for media streams. Rankings suchas “most clicked,” “most commercially successful,” and the like may beprovided, providing feedback and hints to new media streamproviders/content creators on what constitutes a successful set ofassociations. In some embodiments, such rankings are provided in a finergrained form (e.g., most successful commercial associations for aparticular demographic).

As described herein, with respect to the selection of specific items ofsupplemental content to display to given viewers of media streams(perhaps in a given context), any number of factors may be referenced inorder to make this selection. For example, the selection may be based oninformation about a group of viewers (e.g., all viewers in a givengeographic area), potentially as aggregated in a profile of the“typical” viewer or profiles for groups of viewers. This approach wouldenable the presentation of relevant supplemental content withoutnecessitating the computationally burdensome approach of determining theoptimal supplemental content to show to each individual viewer. Otherexamples include determining the selection of supplemental content byreferencing information about the viewing context (e.g., time of day,location), or the viewer's behavior in other mediums or contexts (e.g.,internet searches, retail store purchases as determined throughreferencing activity represented on the viewer's credit card report).Other examples include: the general mood or tone of a given segment ofthe media stream (e.g., sentimental, patriotic, romantic, depressed,hopeful, paranoid); the specific actor or actors in the segment (i.e.,Viewer X loves to watch actress A, particularly when she's appearing ina scene with actor B); type of music (e.g., country, classical, rock,hip-hop); type of action (e.g., violent, sports, dancing, kissing,eating); prevalence of product placements (e.g., Viewer X has neverclicked on a “subtle” product placement); the characteristics of thesegments before the segment in which viewer interest was signaled (e.g.,Viewer X responds well to segments of violence followed by moods ofregret); time of day (e.g., Viewer X doesn't usually watch television atthis time of day, so he/she may have an illness); and supplementalcontent that is selected after referring to data provided by a thirdparty (e.g., weather data for a particular viewer zip code). The variousfactors listed above can be paired with conventional clickstream dataobtained from the viewer's use of the Internet, demographic data (i.e.,zipcode, GPS coordinates), data regarding other viewers within the samehousehold, and the like.

With respect to an automated system to aid the selection of supplementalcontent, as the system becomes more accurate over time, it may continueto be the case that system's statistical models, etc. will never predictperfectly those items of supplemental content that may be most ofinterest to a given viewer. In one embodiment, for a set number orpercentage of the items of supplemental content that are “predicted” tobe of interest to a viewer (i.e., perhaps one percent), the system willselect an item of supplemental content that is either randomlydetermined or is determined to be almost certainly not of interest tothe viewer. Given the sheer number viewer experiences, a small number ofthese “chance” offerings will be selected by a viewer (i.e.,serendipity). In one embodiment, the system will note this serendipity,and will then feed that same item of supplemental content to all otherviewers with similar interests, and in this way may unlock previouslyunidentified “niche” interests.

In one embodiment, whenever a media stream is being displayed, theviewer display device (or an intermediary device connected thereto)periodically looks ahead in the media stream (or across a set of mediastreams being pieced together) by referring to the segments and/or otherinformation provided by the content creator or access provider. Then,whenever the viewer fast forwards through the media stream, any standardsupplemental content associated therewith is replaced with a differentversion of the supplemental content that reads much better at highspeeds (e.g., uses large logos that are on the screen for an extendedperiod of time). This combats the issue of viewers fast-forwardingthrough standard commercial feeds, which can be a problem foradvertisers. In one embodiment, a viewer's history of fast forwardingpast supplemental content is reviewed. If the viewer tends to frequentlyfast forward through supplemental content, then all of the supplementalcontent sent to the viewer will be replaced as described.

In one embodiment, the distributed signal for advertisements (e.g.,television commercials) is actually sent in a first stream and a secondstream. If the advertisement is enabled to play normally, the standard,first stream plays. If the advertisement is fast forwarded, thereplacement second stream plays. Put another way, the fast forwardinstruction simply triggers the cutting back and forth between the firstand second streams.

In one embodiment, the viewer is enabled to set a default setting sothat the fast forwarding initiates automatically, but does not stopautomatically. Thus, the viewer must pay attention to stop the fastforwarding. A preset default would make it easier for replacementcontent to be cached and available.

In one embodiment, the actual period of time over which fast forwardingoccurs is randomly assigned (within a range of parameters) to encouragethe viewer to watch closely as he/she fast forwards (which is why theaverage person has an approximately 21% recall rate of fast forwardedads). This could be accomplished by adding duplicate segments.

The present invention's infrastructure and approach can be used to“hyperlink” virtually the entire world, and to place articles thereinwithin a searchable ontology. In one embodiment, each point of interest(e.g., a particular product display within a store or a particular spotat a national park, zoo or museum) would “register” to be assigned aunique identifier. Then, a website would be created that would enableanyone to create their own website, article feed. Then, the identifieris transmitted continuously (e.g., via ultrasound, Wi-Fi or the like)and received by a viewer's portable device. Thus the viewer can see theinformation immediately or can bookmark the location for review later(e.g., on his/her online account).

In one embodiment, the viewer's device could send the signal necessaryto retrieve the desired information. For example, movie theatres mightsend (e.g., broadcast) a signal so that an audience can bookmark, butnot immediately view, supplemental content associated with a movie beingscreened.

Consider a world in which many articles are tagged with RFIDinformation. For example, all of the articles in a store could becatalogued in a database. Then, assume that the location of a “reader”of RFID tags is known (e.g., the reader is a cellular phone thatincludes integrated GPS capability). Thus, the database could log thelocation of a specific article at a time when the article's tag was lastread. Put another way, the location of the reader is know, and thisknowledge can be used to determine the location of the articleConversely, the location of the article may be known, and this knowledgecould be used to establish the location of the viewer who is reading thearticle. In one embodiment, the viewer's location (and changes andsequences in locations) are tracked by tracking the viewer's interactionwith specific articles for which the database includes location data.This location data may be stored and used as part of the viewer'sprofile with respect to consumer behavior or the like.

In one embodiment, when a reader interrogates an article, the databaseupdates a map of the location of all articles in the database. Then, thereader can make various queries based on those locations and/or variousforms of information can be sent to the reader. For example, if theviewer interrogates an article in a supermarket, the database can tellhim/her the price of the article and the stores to go to in his/hervicinity to find the same article. Alternatively, the viewer can find anarticle on the Internet and then query the database to determine wherethat article may be located.

The viewer device could directly transmit its GPS coordinates andcompare this information with other articles known to be in the area viathe location determination methods described above. This data could becombined with other data known about the viewer such as his/her consumerpreferences (e.g., as determined by past click throughs, purchases,search activity, etc.) or characteristics (e.g., physical measurements)and this information could be used to direct the viewer to specificarticles in his/her physical proximity. For example, the viewer devicemight display a screen with the statement: “You might like the XL bluedress shirt that is ten feet to your left.”

In one embodiment, the viewer device used to access the database couldalso send a signal containing information about the viewer, where theinformation can be used to trigger personalized displays in stores. Inone embodiment, the viewer device contains only the viewer's ID, whichmatches the viewer's profile that is stored in the database. Forexample, the database might include the viewer's income level and aseller might sponsor a customized transaction (e.g., “If you buy theshirt to your left, the store will give you a ten percent discount ifyou show them this screen.”)

Given the number of people in any particular store or on a busy street(e.g., Times Square), it will be impossible to provide each viewer withhis/her own content, given that a large number of people can view thesame content simultaneously. Conversely, it may be unsettling to aviewer if all content he/she receives is too perfectly personalized. Inone embodiment, an abstract of a viewer's demographic and of othercharacteristics of all viewers in a particular area is created, and thisabstract is utilized to select the best advertising for the largestnumber of viewers. For example, Times Square billboards could detect,during the morning hours, that most people in the area are commutingprofessionals (e.g., triggering advertisements for insurance services);that, during the afternoon, most people in the area are tourists (e.g.,triggering advertisements for cheap Broadway tickets); and that, duringthe evening, most people in the area are teenagers from New Jersey(e.g., triggering athletic shoe advertisements featuring their favoritemusic).

It may be the case that pointing an RFID detector of a viewer device ata group of articles (e.g., a collection of articles in a store display)may, because of the lack of specificity in RFID tags, cause a number ofarticles to be triggered in the database, and the viewer will need toselect the article in which he/she is interested. In one embodiment, theviewer interrogates a group of articles, and the database is accessed.The database then sends a list (e.g., words or pictures) of the articlesin the vicinity to the viewer device. The viewer selects the article(s)in which he/she is interested, and the revised query is sent back to thedatabase, which returns more complete information.

In one embodiment, the database is provided with information regardingall articles identified by RFID within a particular location, includinga graphical representation of each article. Then, within a particularlocation (e.g., a supermarket), sensors are placed that can triangulateon (or in some other way determine) the relative location of thearticles within the location. This data may be used to create a virtualworld that simulates the location and the articles found there. Anavatar of the viewer can then “visit” the virtual location, and theadvertising within the location will be customized for the viewer.

In one embodiment, the articles are found within a particular location,but the online virtual location has a different look/feel from theactual location. In other words, the virtual location can be re-arrangedto suit the viewer's preferences. Then, when the viewer visits theactual locations, the GPS capability integrated in the viewer deviceguides the viewer to the actual locations of the articles. For example,a store could be re-arranged for a delivery person who wants to know ifcertain articles need to be restocked.

In one embodiment, using the above-described data, simulations of thelocation layout and the advertising can be run to determine how thelikely viewers in the area would respond to certain articles, prices,advertising displays and the like.

Moreover, although many embodiments of the present invention have beendescribed within the context of media streams distributed via televisionor via the Internet, those skilled in the art will appreciate that theconcepts of the present invention can be applied to other forms of mediastream distribution as well, including movie screenings and video games.For example, each segment of a movie to be screened will be linked witha unique identifier that is mapped to a range of supplemental content. Amovie theater screening the movie may transmit the segment identifierinformation to the entire theatre (e.g., via Wi-Fi or ultrasound), sothat any viewer with a device capable of reading the segment identifiercan capture the segment of interest. This information could provideimmediate supplementation content or could be bookmarked for later(e.g., post-screening) review. Alternatively, if segment-by-segmentidentifier transmission proved impractical, only a single ultrasound orcode identifying the movie could be transmitted, and then timecodescould be used, because there is no potential for commercial interruptionto “throw off” the timing.

In one embodiment, the viewer device is a portable device that can bedocked (for example, in a car) and receive transmissions remotely. Forinstance, the viewer device may be a cellular phone that includes thereader necessary to receive segment identifiers and/or to cause a deviceat which the cellular phone is pointed to send the segment identifier.In the remote access context, each transmitter location (e.g., inmuseums, national parks, malls and the like) is described along with avariety of parameters (e.g., outdoors or indoors, scenic or not scenic,urban or rural, type of information conveyed (for example, a scenic viewof a waterfall in Yosemite park or a billboard on the side of the NewJersey Turnpike) and parameters (color, size, content, approach etc.) ofconnected advertising content). This description provides usefulinformation as to the viewer's behavior, reactions and the like.

In one embodiment, the transmitter description input is matched toadditional information available from other sources. For example, it maybe possible to know the general weather conditions at a given time andlocation. As another example, the viewer might be responding to ahighway billboard, and it would be useful to know the speed of theviewer's car (or the general traffic speed), as this might suggest thatthe billboard is effective only at certain speeds.

In another embodiment, the concepts of the present invention are appliedin the context of a computer or video game. Each location and article ofrelevance within a particular game may be assigned a unique identifierto the segment identifier typically used. The database would storeinformation about the locations or articles (or the combinations of thelocations and articles) that is similar to the segment classificationdata (e.g., type of environment, type of game, font used, image used,etc.) Further, the avatars (i.e., the computer embodiments of thegamers) in proximity to the supplemental content space in the game wouldalso be reported. Then, the same approaches described above are appliedin the game environment (e.g., determining the “optimal” supplementalcontent to feed to that location given the demographic mix andpreferences of viewers gathered at that location). The supplementalcontent could include billboards, live videos embedded in thebackground, articles left on the street or in a store window that theavatar could click on (e.g., a specific automobile that the avatardrives in a video game) and the like. The advantages of using thisapproach are the same as for video: (1) the supplemental content neednot be embedded into the game (for those games that aren't “hosted”centrally); (2) the supplemental content can be targeted to theindividual viewer/player; (3) the service provides an entireinfrastructure for the supplemental content; and (4) the targeting isable to leverage data that is generated by the viewer's/player'sinteractions with other media (e.g., what the viewer has “clicked” withrespect to advertising in the video context).

Furthermore, as described above, the concepts of the present inventionmay be advantageously applied for the purposes of data mining. Forexample, the unique statistics captured by segment identifiers, viewerdemographics, past viewer responses and the like may support data miningfor any number of purposes that are unrelated to the viewer's directinteraction with media streams. Put another way, the database mapped tosegment identifiers has the potential to translate many differentaspects of a given segment into various numerical information.Similarly, viewer interactions can be translated into numericalinformation. From this basis, statistical modeling can be utilized topredict the response that a given media stream is likely to generatefrom a given group (or, conversely, the type of content to which thatgiven group is more likely to respond positively).

In one embodiment, a number of templates of proxy media streams arecreated, where the proxy media streams simulate different types of mediastreams. This may help advertisers in determining the types of mediastreams in which they should place articles or supplemental contentrelated thereto. Once these statistics are obtained, actual mediastreams may also be examined prior to viewing. In this way, a biddingsystem can be arranged to auction advertising space and the like.

FIGS. 47 and 48 illustrate another application of aspects of the presentinvention. Specifically, as illustrated in FIG. 47, a viewer viewing amedia stream on a home theater viewing device 4700 may choose to pausethe media stream and send the segment identifier identifying the currentsegment of the media stream to his/her account with an access provider4702. As illustrated in FIG. 48, the viewer may later request to receivethe media stream (potentially via a new display device 4800, such as aportable media player), resuming at the segment at which he/she pausedthe media stream. The access provider 4702 would then read the segmentidentifier stored in the viewer's account to retrieve the requestedsegment(s) of the media stream and distribute the requested segment(s)to the viewer.

FIG. 13 is a high level block diagram of the media stream annotationmethod that is implemented using a general purpose computing device1300. In one embodiment, a general purpose computing device 1300comprises a processor 1302, a memory 1304, an annotation module 1305 andvarious input/output (I/O) devices 1306 such as a display, a keyboard, amouse, a modem, and the like. In one embodiment, at least one I/O deviceis a storage device (e.g., a disk drive, an optical disk drive, a floppydisk drive). It should be understood that the annotation module 1305 canbe implemented as a physical device or subsystem that is coupled to aprocessor through a communication channel.

Alternatively, the annotation cancellation module 1305 can berepresented by one or more software applications (or even a combinationof software and hardware, e.g., using Application Specific IntegratedCircuits (ASIC)), where the software is loaded from a storage medium(e.g., I/O devices 1306) and operated by the processor 1302 in thememory 1304 of the general purpose computing device 1300. Thus, in oneembodiment, the annotation module 1305 for mapping metadata to mediastreams described herein with reference to the preceding Figures can bestored on a computer readable medium or carrier (e.g., RAM, magnetic oroptical drive or diskette, and the like).

Thus, the present invention represents a significant advancement in thefield of multimedia distribution and advertising. Embodiments of theinvention enable content creators, advertisers and other parties to markup media streams such as television shows, commercials, infomercials,trailers, shopping channel segments, music videos, movies, VHS or DVDrecordings, video-on-demand content, video games (e.g., interactivetelevision games, personal computer games, online games, handheld gamesand console-based games), online hosted video and the like withsupplemental content relating to articles that appear in the mediastreams (e.g., commercial articles, actors, songs, etc.). Thus,information and advertising content (such as product placements) may beimplemented in a manner that is computationally simplified from atechnical perspective and substantially unobtrusive from a viewerperspective.

It should be noted that although not explicitly specified, one or moresteps of the methods described herein may include a storing, displayingand/or outputting step as required for a particular application. Inother words, any data, records, fields, and/or intermediate resultsdiscussed in the methods can be stored, displayed, and/or outputted toanother device as required for a particular application. Furthermore,steps or blocks in the accompanying Figures that recite a determiningoperation or involve a decision, do not necessarily require that bothbranches of the determining operation be practiced. In other words, oneof the branches of the determining operation can be deemed as anoptional step.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred embodiment shouldnot be limited by any of the above-described exemplary embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A method for distributing an annotated mediastream created by a first user, comprising: selecting a media streamunder instruction from the first user, wherein the media stream isassociated with an index of one or more segment identifiers that areassociated with one or more segments of the media stream; selecting atleast one item of supplemental content under instruction from the firstuser; mapping the at least one item of supplemental content to at leastone of the one or more segment identifiers, such that the media streamremains substantially unaltered, wherein the mapping is performed underinstruction of the first user to produce a mapping; storing the mediastream and the mapping; receiving a first request from a second user,different from the first user, to view the media stream; distributingthe media stream and the mapping over a network to a first displaydevice operated by the second user, wherein the media stream and themapping produce the annotated media stream when utilized together, suchthat the at least one item of supplemental content and a segment of theone or more segments of the media stream indicated by the one or moresegment identifiers are displayed simultaneously in a time-synchronizedmanner such that the at least one item of supplemental content relatesto a localized context specific to the segment, where the at least oneitem of supplemental content and the segment are displayed in separate,non-overlapping viewing panes; separating the at least one item ofsupplemental content from the media stream; sending the at least oneitem of supplemental content, independent of the media stream, to asecond display device operated by the second user for later viewingwithout requiring an input from the second user or from the seconddisplay device; collecting a commission when the annotated media streamis viewed by the second user; providing a first portion of thecommission to the first user, wherein an amount of the first portion ofthe commission is calculated, at least in part, based on a number oftimes the annotated media stream has been viewed; and providing a secondportion of the commission to one or more individuals who selected the atleast one item of supplemental content for mapping, wherein anallocation of the second portion of the commission among the one or moreindividuals is dictated by the one or more individuals.
 2. The method ofclaim 1, wherein the at least one item of supplemental content relatesto an article, a person, or a concept, and the at least one item ofsupplemental content comprises at least one of: text, audio, video, amultimedia presentation, a photograph, an image, a uniform resourcelocator, a computer application, data, commentary, advertising copy, akeyword, a tag, an element that encourages interactive participation bythe viewer, variable data, raw data, stylized information, or aplaceholder.
 3. The method of claim 2, wherein the article, person orconcept is depicted in or suggested by the media stream.
 4. The methodof claim 1, wherein the at least one item of supplemental content isdisplayed after the media stream has concluded.
 5. The method of claim1, further comprising: receiving a first signal from the second user inresponse to the at least one item of supplemental content.
 6. The methodof claim 5, further comprising: logging the first signal.
 7. The methodof claim 6, further comprising: reporting the first signal to at leastone of: the first user or a provider of the at least one item ofsupplemental content.
 8. The method of claim 5, further comprising:providing information to the second user in response to the firstsignal, the information relating to the at least one item ofsupplemental content.
 9. The method of claim 6, further comprising:logging a response of the second user to the information provided as asecond signal.
 10. The method of claim 9, further comprising: providingthe second signal to at least one of: the first user or a provider ofthe at least one item of supplemental content.
 11. The method of claim1, wherein the annotated media stream comprises at least one of: a videocomponent, an audio component, or a text component.
 12. The method ofclaim 1, wherein the at least one item of supplemental content isselected from among a plurality of potential items of supplementalcontent mapped to the at least one of the one or more segmentidentifiers, a selection of the at least one item of supplementalcontent being based at least in part on information relating to thesecond user.
 13. The method of claim 12, wherein the informationrelating to the second user comprises at least one of: demographicinformation about the second user, information about past viewing habitsof the second user, information about past interactions the second userhas had with other items of supplemental content, a number of times thesecond user has viewed the annotated media stream, an average time forwhich the second user has viewed other media streams, second userpreferences as expressed by the second user and learned second userpreferences.
 14. The method of claim 1, wherein the at least one item ofsupplemental content is selected from among a plurality of potentialitems of supplemental content mapped to the at least one of the one ormore segment identifiers, a selection of the at least one item ofsupplemental content being based at least in part on a device on whichthe annotated media stream is displayed.
 15. The method of claim 1,wherein the at least one item of supplemental content is randomlyselected from among a plurality of potential items of supplementalcontent mapped to the at least one of the one or more segmentidentifiers.
 16. The method of claim 1, wherein the at least one item ofsupplemental content is selected from among a plurality of potentialitems of supplemental content mapped to the at least one of the one ormore segment identifiers, a selection of the at least one item ofsupplemental content being based on a rotating schedule.
 17. The methodof claim 16, wherein after cycling through the rotating schedule anumber of times, the selection is based at least in part on a responseof the second user to the plurality of potential items of supplementalcontent.
 18. The method of claim 17, wherein the selection is based atleast in part on which of the plurality of items supplemental ofsupplemental content generated a highest degree of viewer interest. 19.The method of claim 1, wherein the at least one item of supplementalcontent is selected from among a plurality of potential items ofsupplemental content mapped to the at least one of the one or moresegment identifiers, a selection of the at least one item ofsupplemental content being based at least in part on a context in whichthe displaying takes place.
 20. The method of claim 1, wherein the atleast one item of supplemental content is selected from among aplurality of potential items of supplemental content mapped to the atleast one of the one or more segment identifiers, a selection of the atleast one item of supplemental content being based at least in part on ageneral tone of the at least a portion of the media stream.
 21. Themethod of claim 1, further comprising: displaying banner advertisingwith the displayed annotated media stream on a common display device,where content of the banner advertising is selected in accordance withat least one of: a keyword associated with the annotated media stream, atarget demographic of the annotated media stream, the at least one itemof supplemental content, a position of the at least one item ofsupplemental content within a hierarchy of a library of supplementalcontent, or information about the second user.
 22. The method of claim1, wherein the media stream and the at least one item of supplementalcontent are retrieved from different sources prior to the displaying.23. The method of claim 1, further comprising: detecting that the seconduser is fast forwarding through the annotated media stream; andreplacing the at least one item of supplemental content with areplacement item that is more easily viewable during fast forwarding.24. The method of claim 1, wherein the selecting the media streamcomprises: receiving the media stream from the first user over anetwork.
 25. The method of claim 1, wherein the separate,non-overlapping viewing panes are displayed on a common display.
 26. Themethod of claim 25, wherein a viewing pane of the separate,non-overlapping viewing panes in which the segment of the media streamis displayed has a frame size that remains substantially consistentthroughout a display of the media stream.
 27. A non-transitory computerreadable medium containing an executable program for distributing anannotated media stream created by a first user, where the programperforms steps of: selecting a media stream under instruction from thefirst user, wherein the media stream is associated with an index of oneor more segment identifiers that are associated with one or moresegments of the media stream; selecting at least one item ofsupplemental content under instruction from the first user; mapping theat least one item of supplemental content to at least one of the one ormore segment identifiers, such that the media stream remainssubstantially unaltered, wherein the mapping is performed underinstruction of the first user to produce a mapping; storing the mediastream and the mapping; receiving a first request from a second user,different from the first user, to view the media stream; distributingthe media stream and the mapping over a network to a first displaydevice operated by the second user, wherein the media stream and themapping produce the annotated media stream when utilized together, suchthat the at least one item of supplemental content and a segment of theone or more segments of the media stream indicated by the one or moresegment identifiers are displayed simultaneously in a time-synchronizedmanner such that the at least one item of supplemental content relatesto a localized context specific to the segment, wherein the at least oneitem of supplemental content and the segment are displayed in separate,non-overlapping viewing panes; separating the at least one item ofsupplemental content from the media stream; sending the at least oneitem of supplemental content, independent of the media stream, to asecond display device operated by the second user for later viewingwithout requiring an input from the second user or from the seconddisplay device; collecting a commission when the annotated media streamis viewed by the second user; providing a first portion of thecommission to the first user, wherein an amount of the first portion ofthe commission is calculated, at least in part, based on a number oftimes the annotated media stream has been viewed; and providing a secondportion of the commission to one or more individuals who selected the atleast one item of supplemental content for mapping, wherein anallocation of the second portion of the commission among the one or moreindividuals is dictated by the one or more individuals.
 28. System fordistributing an annotated media stream created by a first user,comprising: a first user device for selecting a media stream and atleast one item of supplemental content under instruction from the firstuser, wherein the media stream is associated with an index of one ormore segment identifiers that are associated with one or more segmentsof the media stream, and wherein the first user device is also formapping the at least one item of supplemental content to at least one ofthe one or more segment identifiers, such that the media stream remainssubstantially unaltered, wherein the mapping is performed underinstruction of the first user to produce a mapping; a server for storingthe media stream and the mapping, the server being further configuredfor receiving a first request from a second user device, different fromthe first user device, to view the media stream and for distributing themedia stream and the mapping to the second user device over a network,wherein the media stream and the mapping produce the annotated mediastream when utilized together, such that the at least one item ofsupplemental content and a segment of the one or more segments of themedia stream indicated by the one or more segment identifiers aredisplayed simultaneously in a time-synchronized manner such that the atleast one item of supplemental content relates to a localized contextspecific to the segment, where the at least one item of supplementalcontent and the segment are displayed in separate, non-overlappingviewing panes, the server being further configured for separating the atleast one item of supplemental content from the media stream and sendingthe at least one item of supplemental content, independent of the mediastream, to a third user device different from the first user device andthe second user device without requiring a user input or a signal fromthe third user device and wherein the server is also for collecting acommission when the annotated media stream is viewed on the second userdevice and for providing a first portion of the commission to the firstuser and a second portion of the commission to one or more individualswho selected the at least one item of supplemental content for mapping,wherein an amount of the first portion of the commission is calculatedbased at least in part on a number of times the annotated media streamhas been viewed, and an allocation of the second portion of thecommission among the one or more individuals is dictated by the one ormore individuals.
 29. A method for distributing an annotated mediastream created by a first user, comprising: selecting a media streamunder instruction from the first user, wherein the media stream isassociated with an index of one or more segment identifiers that areassociated with one or more segments of the media stream; selecting atleast one item of supplemental content under instruction from the firstuser; mapping the at least one item of supplemental content to at leastone of the one or more segment identifiers, such that the media streamremains substantially unaltered, wherein the mapping is performed underinstruction of the first user to produce a mapping; distributing themedia stream and the mapping to a first display device operated by asecond user, different from the first user, wherein the media stream andthe mapping produce the annotated media stream when utilized together,wherein the distributing comprises displaying the at least one item ofsupplemental content and at least a portion of the media stream in asimultaneous, time-synchronized manner such that the at least one itemof supplemental content relates to a localized context specific to thesegment, wherein the at least one item of supplemental content and asegment of the one or more segments of the media stream indicated by theone or more segment identifiers are displayed in separate,non-overlapping viewing panes; separating the at least one item ofsupplemental content from the media stream; sending the at least oneitem of supplemental content, independent of the media stream, to asecond display device operated by the second user for later viewingwithout requiring an input from the second user or from the seconddisplay device; collecting a commission when the annotated media streamis viewed by the second user; providing a first portion of thecommission to the first user, wherein an amount of the first portion ofthe commission is calculated, at least in part, based on a number oftimes the annotated media stream has been viewed; and providing a secondportion of the commission to one or more individuals who selected the atleast one item of supplemental content for mapping, wherein anallocation of the second portion of the commission among the one or moreindividuals is dictated by the one or more individuals.
 30. The methodof claim 29, wherein the collecting requires further action on a part ofthe second user.
 31. The method of claim 30, wherein the further actionis a selection of a hyperlink contained in the at least one item ofsupplemental content.
 32. The method of claim 30, wherein the furtheraction is a consummation of a commercial transaction.
 33. The method ofclaim 29, further comprising: providing at least a portion of thecommission to a contributor of the at least one item of supplementalcontent.
 34. The method of claim 29, wherein the commission is amonetary commission.
 35. The method of claim 29, wherein the commissionis in a form of a redeemable coupon or credit.
 36. The method of claim29, wherein the commission is collected from a provider of the at leastone item of supplemental content.
 37. The method of claim 29, whereinthe commission is collected from a seller of an article represented bythe at least one item of supplemental content.
 38. The method of claim29, wherein the at least one item of supplemental content relates to anarticle, a person, or a concept, and the at least one item ofsupplemental content comprises at least one of: text, audio, video, amultimedia presentation, a photograph, an image, a uniform resourcelocator, a computer application, data, commentary, advertising copy, akeyword, a tag, an element that encourages interactive participation bythe viewer, variable data, raw data, stylized information, or aplaceholder.
 39. The method of claim 38, wherein the article, person orconcept is depicted in or suggested by the media stream.
 40. The methodof claim 29, wherein an amount of the commission is dictated by aprovider of the commission prior to the second user viewing theannotated media stream.
 41. The method of claim 29, wherein theselecting the media stream comprises: receiving the media stream fromthe first user over a network.
 42. The method of claim 29, wherein theseparate, non-overlapping viewing panes are displayed on a commondisplay of a device used by the second user.
 43. The method of claim 42,wherein a viewing pane of the separate, non-overlapping viewing panes inwhich the at least a portion of the media stream is displayed has aframe size that remains substantially consistent throughout a display ofthe media stream.
 44. A non-transitory computer readable mediumcontaining an executable program for distributing an annotated mediastream created by a first user, where the program performs steps of:selecting a media stream under instruction from the first user, whereinthe media stream is associated with an index of one or more segmentidentifiers that are associated with one or more segments of the mediastream; selecting at least one item of supplemental content underinstruction from the first user; mapping the at least one item ofsupplemental content to at least one of the one or more segmentidentifiers, such that the media stream remains substantially unaltered,wherein the mapping is performed under instruction of the first user toproduce a mapping; distributing the media stream and the mapping to afirst display device operated by a second user, different from the firstuser, wherein the media stream and the mapping produce the annotatedmedia stream when utilized together, wherein the distributing comprisesdisplaying the at least one item of supplemental content and at least aportion of the media stream in a simultaneous, time-synchronized mannersuch that the at least one item of supplemental content relates to alocalized context specific to the segment, wherein the at least one itemof supplemental content and a segment of the one or more segments of themedia stream indicated by the one or more segment identifiers aredisplayed in separate, non-overlapping viewing panes; separating the atleast one item of supplemental content from the media stream; andsending the at least one item of supplemental content, independent ofthe media stream, to a second display device operated by the second userfor later viewing without requiring an input from the second user orfrom the second display device; collecting a commission when theannotated media stream is viewed by the second user; providing a firstportion of the commission to the first user, wherein an amount of thefirst portion of the commission is calculated, at least in part, basedon a number of times the annotated media stream has been viewed; andproviding a second portion of the commission to one or more individualswho selected the at least one item of supplemental content for mapping,wherein an allocation of the second portion of the commission among theone or more individuals is dictated by the one or more individuals. 45.System for distributing an annotated media stream created by a firstuser, comprising: a first user device for selecting a media stream andat least one item of supplemental content under instruction from thefirst user, wherein the media stream is associated with an index of oneor more segment identifiers that are associated with one or moresegments of the media stream, and wherein the first user device is alsofor mapping the at least one item of supplemental content to at leastone of the one or more segment identifiers, such that the media streamremains substantially unaltered, wherein the mapping is performed underinstruction of the first user to produce the annotated media stream; anda server for distributing the media stream and the mapping to a seconduser device, different from the first user device, wherein the mediastream and the mapping produce the annotated media stream when utilizedtogether, wherein the distributing comprises displaying the at least oneitem of supplemental content and at least a portion of the media streamin a simultaneous, time-synchronized manner such that the at least oneitem of supplemental content relates to a localized context specific tothe segment, wherein the at least one item of supplemental content and asegment of the one or more segments of the media stream indicated by theone or more segment identifiers are displayed in separate,non-overlapping viewing panes, the server being further configured forseparating the at least one item of supplemental content from the mediastream and sending the at least one item of supplemental content,independent of the media stream, to a third user device different fromthe first user device and the second user device without requiring aninput from the second user or from the third user device, and whereinthe server is also for collecting a commission when the annotated mediastream is viewed on the second user device and for providing a firstportion of the commission to the first user and a second portion of thecommission to one or more individuals who selected the at least one itemof supplemental content for mapping, wherein an amount of the firstportion of the commission is calculated based at least in part on anumber of times the annotated media stream has been viewed, and anallocation of the second portion of the commission among the one or moreindividuals is dictated by the one or more individuals.